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"^That man, I thiik, has had a liberal education who 
has been so trained in youth that his body is the ready ser- 
vant of his will, and does with ease and pleasure all the work 
that, as a mechanism, it is capable of ; whose intellect is a 
clear, cold , logic engine, with afll its parts of equal strength, 
and in smooth working order ; ready, like a steam engine, 
to be turned to any kind of work, and spin the gossamers 
as w'ell as forge the anchors of the mind ; whose mind is^ 
stored with a knowledge of the great and fundamental truths 
of Nature and of the laws of her operations ; one who, no 
stunted ascetic, is full of life and fire, but whose passions are 
trained to come to heel by i vigorous will, the servant of 
a tender conscience ; who has learned to love all beauty, 
whether of Nature or of art, to hate all vileness, and to 
resjfcct others as himself. 

" Such an one and no other, I conceive, has had a liberal 
education." — Huxley, Science and Education, p. 86. 
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INTRODUCTION 

In the first paper in this little book, delivered as 
a lecture at the Royal Society of Arts in January 
1911 (and published in a first edHion by Hugh 
^ees), I put forward a plea, which received the 
powerful support of Lord Cioner, for a Royal 
Commission to inquire into the working of the 
examination system. The paper was discussed 
in a number of leading articles in the press. I 
cannot say what influence the paper itself, ^r 
Lord Cromer's views, or public comments, had 
on the Government. At any rate, in March 1912, 
a Royal Commission, presided over *6y Lord 
MacDonri^U, was appointed with the following 
reference : 

To inquire into and report on the methods 
of making appointments to and promotions 
iif the Civil Service, including the Diplomatic 
and Consular Services, apd the legal depart- 
ments ; 

To investigate the working and efficiency 
of the system ^f competitive examination 
ix 
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for such appointments, and to make recom- 
mendations for any alterations or improve- 
ments in that sysj^m which may appear to 
be advisable ; and 

To consider whether the existing scheme 
o# organisation meets the requirements of 
the Public Service, and to suggest any modifi- 
caKqps whi^ may be needed therein. 

In April 1914 (after three Reports of a formal 
uature accompanying t|he issue of volumes of 
evidence) the Commission published its* Fourth 
Report (with a Minority Report) in a volume 
extending to over 150 folio pages. ^ The last 
paragraph of the Majority Report reads as follows : 

In submitting this Report on the first part of 
our reference we desire,*' etc. — here follows a 
well-deserved tribute to the work of the Secre- 
taty. 

Now the Report, as a matter of fact, deals 
not only, with the first, but with the third part 
of the reference, and contains important sugges- 
tions for the reorganisation of the Civif Service. 
But with the second part of the reference — the 
working and efficiency of the examination system 
— the Commission neither profess to have deali^ 
nor have they dealt, except in a most fragmentary 
fashion, and it is important to bring out this fact 
lest it be supposed that the work which the ^ 
Commission were asked to do in re*gard to examina- 

* Royal Com mission on the Civil Service [Cd, 7338]. 
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INTRODUCTION 

tiODs has, iri fact, been carried out. I quo‘:e m 
Note A, pp. 80-88 bebw, the main passages of 
the Report dealing witi^ the working nf the 
examination system. 

I have been unable to discover in the whole of 
the fourth Report any coiitributbti out 
knowledge of the working of that system ml 
examinations, or any suggestion ^ to itsjiftprove 
ment. The Commission say that they propose 
in due place '' [see Note A^p. bx] to examine 
the defects attaching to%the s^^stem, and suggest 
that they are susceptible of some mitigation.' 
Blit the examination of these defects is reserved 
for a further Report, of whbh, perhaps owing 
to the war, there has been no sign. In their 
formal Recommendations^ the Commission say 
that examinations should be directed to testing, 
inter alia, the formation of character ^ ; but 
they give no indication of how this difficult t&t 
is to be carried out. I have no desire to criticise 
the Royal Commission in this matter ; wish to 
point out that tVey have furnished no matter 
for criticism. It would be convenient to know 
whether the Commission have any intention of 
completing the investigation with which they 
were entrusted. A large body admirably repre- 
senting many important interests, but not par- 
ticularly well acquainted with the examination 
4 system, may well have recoiled before an expert 
and ungrateful task of this kind, and have rested 

1 See Report, p. for, quoted in Note A, p. 88. 
xi 
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content with the weighty conclusions they have 
reached in regard to other matters. But it is 
a fact that ignorance/of the real meaning and 
working of our examinations remains in 1917 
what it was in 1911 ; and the plea for a com- 
J>rehe 5 Lisiv^ scientific investigation holds as good 
nrDw as it did then. 

It v^yld, however, be wrong not to point out 
tfi.e very fruitful result to which one of the sug- 
gestions of the Majority Report of the Commis- 
sion ^ has led in the shape of the Report of Mr. 
Stanley Leathes* Committee on Civile Service 
Examinations. If the recommendations oftthe 
Committee are adopted, they will mark a neW 
era in our system of competitive examinations 
by the importance th^y will give to that part of 
examinations for which cramming is impossible, 
and they will go far to carry out certain sugges- 
tfbns which I have made in regard to this subject.* 

The object of the second paper in this volume, 
originally published in The Times Educational 
Supplement of May loth, 1917, though it covers 
somewhat the same ground as the first, is‘ different.* 

^ See Majority Report, § 50, quoted on p. 87, below. 

* For details in regard to the personnel and reference 
the Committee and a summary of their recommendations, 
see Note B, p. 89 below. 

® The paper, as actually written, though composed for 
another purpose, was submitted to Mr. Stanley Leathes’ 
Committee at the request of one of the members ; it 
not printed quite in exienso in the Educational Supplement, 
I have made a few modihcationscof it for its republication. 
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INtRODUCTICN 

It is intended to give a more general perspec.tiV6 
of the subject as a whole, though in last para- 
graph I have not been a lie to resist thi tempta- 
tion to appeal once njore to examining bodies 
to tell us in clear and oiim^stakable terms the 
meaning of thrir various examination Certificate!. 
The paper may be regarded as a series of chapter 
headings for a larger work whi#i T ha^re had hi 
mind but which I have net had the leisure to write. 
The subject, important in 1911, is even more 
important now Mr. Fisher’s remarkable speech^ 
as President of the Board of Education, and his 
Education Bill of this year, seem to show that the 
Government are in earnest about educational 
reconstruction, that the long campaign of enlight- 
enment and propaganda in which Mr. M. E. 
Sadler has taken the chief part is at last about 
to bear fruit. The working classes and their 
leaders, whose share in the government of fhe 
country is rapidly becoming of increasing — I was 
almost saying predominant — importanft, are no 
less in earnest. But in dealing with examina- 
tions there is still timidity of thought and action 
that may go far to wreck our new-born hopes. 

Let it not be said that examinations do not 
matter. Every teacher, every administrative 
educational body, knows that, whatever be the 
schemes and curricula, examinations do effec- 
tively control the class-rooms of our Secondary 
Schools, and that they control to a considerable 
extent our Universitaes also ; and no educational 
xiii 
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reform which leaves our examination system as it 
is, will or can be a real reform. Professor John 
Burnet, the distinguisl^d philosopher and Dean 
of the Faculty of Arts at St. Andrews, has written 
an interesting book on Higher Education the 
War}* There are few pages in it, after the initial 
clfapters on '' Kultur and Humanism '' that do 
not refer#to exanninations, directly or indirectly. 
Yet, speaking (on page 195) of classical teaching 
Professor Burnet declares that : The written 
part of the examination «an only deal successfully 
with that part of it which is destined id be for- 
gotten as soon as it has served ^its purpose, and it 
can tell us next to nothing of what is to remain 
as a possession for ever.” I do not disagree with 
Professor Burnet. Orv the contrary, I wish he 
had travelled with me further. And in this con- 
nection I want to plead for the retention of our 
Efiglish word ” culture,” which is in so great a 
danger of being ostracised through the recent 
degeneriHion of its German gousin Kultur} I 
want the word especially because it enables me 
to summarise briefly my view of examinations 
in relation to education as a whole. Using the 
word as I have myself used it in the text, or as 
Professor A. N. Whitehead has recently defined it** 

^ Macmillan & Co., 1917. 

* See Note C, p. 93, below. 

» In his original and stimulating series of essays entitled, 
The Organisation of Thought (Williams & Norgate, 1917), 
p. 8 and passim. * 
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IKTRODTJCTlO'j 

— *' Activity of thought, and receptivencss to 
beauty, and humane feehng ” — it 
that vital part of eiucalion which i.annc-t be tested 
by the ordinary written ' examination. Examina- 
tions test, and can test, tha^ pait of a man's 
(or a womaDs) education in which it is useful 
and necessary to think and act more or Jess Ai 
a model, and which I have cduciition lor 

technical ehiciency. Culture is that part of 
education which is meaningless if it is not sensi- 
tive and individual. * 

It is convenient to divide education into educa- 
tion for technical efficiency and education for 
culture, remembering, howeve*, that efficiency in 
its widest sense implies culture also. The social 
fitness of an individual depends not on one of 
these factors alone but on both, and both must 
be considered in devising our new schemes of edj^- 
cation. My complaint of reformers like Professor 
Burnet is that their criticism of the much-abused 
examination system leaves it as supremely un- 
reformed .as ever. And I urge two correlative 
reforms : first, that examinations should be made 
much more real and drastic tests of technical 
efRciency than they are at present ; and, secondly, 
that a ^arge portion of the educational field should 
be entirely freed from their influence. In what 
follows I have given my reasons for regarding 
many of our present examinations as insufficiently 
searching in matters recognised as of the first 
importance ; but I inay give an example here of 
b XV 
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the use which should be made, in my judgment, 
of that ^portion of th^ school time-table which *1 
hope some day to see freed from examination 
control altogether. I should like to see every 
boy a.nd.girl who intends to follow a scfentific 
career obliged to attend classes in literature and 
history with nj examination on these subjects, 
the teacher being forced either to interest his 
pupils or to suffer from their boredom, without 
.any examination whig at his command, and 
I should like to see every boy or girl whp intends 
to follow an ** Arts career^ obliged to attend 
classes in natural science of the same kind. What 
a revolution we should have in the teaching of 
these non-examination classes ! What a blessing 
for the good teacher*; what a stimulus for the 
poor one ! I desire here to protest emphatically 
aigainst the idea of Mr. R. W. Livingstone, ex- 
pressed in his recent book,* that science is ‘*un- 
imagin<^tive.'' It is quite true that elementary 
science can be extraordinarily dull and inhuman. 
So (as he sees) can elementary history, or Latin 
accidence ; they are so in the hands of teachers 
who have neither historical training nor philo- 
sophic insight. The same holds good for 
science, and one is bound to admit that much 
of the teaching of science ** has been in the 

^ R. W. Livingstone — A Defence of Classical Education, 
1916, [Some of the attacks on Mf. Livingstone's boolc, 
which seems to me in many ways admirable, are not justified 
by the text, in which he claims fbr science a great share in 
education {cf, pp. ^1 et 5^.)]. 
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past all that Mr. Livingstone depicts iS-uii* 
critical, imhistorical, unscientific, ^inavvare <^hat 
our scientific description of nature ^onsUntly 
changing v;ith the progress of science, or that 
the uchievf^ments of science arc tuaiked by the 
individualities ci great men as certainly .tis the 
achievenients of lireratun. Even in matKe- 
matics, the most compelling all sciwaces, 
presentation of truths is nc‘t a unique presenta- 
tion which imposes itself on all alike ; there are 
many presentations possible, and our traditional 
methods, now so largely changing, are character- 
ised by the individualities of the great mathe- 
maticians of the past. While *t is true that in 
literature and in art there can be no compulsion 
of the individual to admire or accept anything 
(although pro fanto, we must be content to be 
regarded as barbarians if we refuse our admira- 
tion to certain of the greatest" men); and, 
on the contrary, in science the compulsion in 
regard to some iundamental truths is 'lorced on 
us by omr physical environment, yet there is a 
great aesthetic and individual side in science, of 
which, from pure ignorance, many literary men 
fail to appreciate the existence. The terms " a 
beautiful theorem," " a beautiful experiment," 
"a beautiful theory," correspond to beauty as 
real to those who feel it as the beauty of a poem 
or a picture. The " wonders of science " com- 
monly brought fomard for our admiration are 
small things compared with the evolution of the 
xvii 
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Newtonian theory of gravitation from the obser- 
vations of Kepler, or of the evolution of Ampere's 
theory o*f electro-dynamics from a few simple 
experiments of Oerstedi But the culture of 
science, like the culture of art, is not for the 
e xamihati6n-room . 

•If the problems of examination touch the 
general pfoblem?of education at every point they 
nevertheless require separate investigation and 
treatment ; and it is for this investigation that I 
]i)lead, and to which I hbpe that this little book 
may be regarded as a contribution. Will the 
new Secondary Schools Examination Council 
created by Mr. Fisher tread the old paths ? Or 
will they avail themselves of a really great and 
new opportunity ? 

August 1917. 


P. J. H. 


©wing to absence abroad the author has been unable 
to see this book through the press, and he desires to ex- 
press his sincere thanks to Professor T. P. Nunn and to 
Mr. ArthufiVatson, who have most kindly revised the proofs 
for him, and also to Professor F. Y. fedgeworth for his help 
in the revision of Note E. * 


[The two following essays are based in part not only on pre- 
vious publications by the writer, but also on a joint article 
on ‘ ' Examinations " by Mr. Arthur Watson and himself in the 
eleventh edition of tlie Encyclopedia Britannica, whldi con- 
tains a fairly extensive bibliography of the subject. The 
author desires, further, to refer his readers to the article on 
Examinations signed by Prof. A. N. Whitehead as Chair- 
man of the relevant committee of the Education Reform 
Council," and published in the valuable collection of Reports 
entitled Education Reform (P. S. King & Son, 1917).] 
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EXAMINATIONS AND THEIR 
RELATION T O CUIH’UltE 
AND EFFICIENCY.' 


I 

EXAMINATIONS In’ THEIR BEARING 
ON NATIpNAL EFFICIENCY 

(A lecture delivered at the Royal Society of Arts in 
Jameary 1911.) 

May I begin with a personal explanation ? What 
I have to say to-night is said on my own behalf 
alone, and no one else in the University which I 
have the honour to serve must be regarded as 
committed in any way to agreement either with 
my main thesis or.with any incidental opinions I 
may expjess. 

To avoid all possible misunderstanding I shall 
state my position at the outset. I do not desire 
to appear as counsel either for the plaintiff or 
for the defendant in the great controversy on 
examinations wliich has raged intermittently 
for more than half a century in this country, as 
in other countries, but rather as one of the public, 
deeply interested in the issues, and trying to see 
clearly through the ffeat and dust of the conflict. 

B 
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It is often said, and generally accepted, that 
the whole of English education is controlled by 
the examination system. It is well, however, in 
dealing with educational questions, to keep some- 
jvhere in the back of one's mind the facjt that 
there^is another kind of education than the educa- 
tion of the schools, the education that Mr. Samuel 
Weller'^ father^ gave him when he let him have 
the run of the streets and make shift for himself ; 
well to remember that there is one branch of the 
national activity, not'' the least successful or 
important — 1 mean its commerce — which up to 
the present has been but little ^uected by examina- 
tion-ideals ; and that our agriculture and indus- 
tries have been artected by them only to a minor 
extent. Our business*, our agriculture, our indus- 
tries, have to a large extent “ sliifted for them- 
selves/' I will not say without the help of school 
teaching, but very largely without the help of 
examinations. 

But t'he public demands that persons on whose 
services it relies, and for whose failures it cannot 
be compensated, as by a business man who fails 
to fulfil his contract — that these should produce 
some certificate of competency based on an 
examination, and often on a series of e;x3.niina- 
tions beginning in childhood and prolonged into 
early manhood and beyond. Teachers, lawyers, 
doctors, dentists, engineers, a^cliitects, and th^ 
civil, naval, and military servants of the Crown, 
on whom the organisation and defence of the 
2 
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Empire rest, must pass examinc.tion tests aud 
ii) many cases their whole career is deterained by 
such tests. 

It is, therefore, a rial ter of national and 
imperial concern that thir mett-od cf test shcluld 
be wise in iO". detig(i, certaui and not Caprlcions 
m it: working, and, above all, that in its opert^ 
tions it shauid not damage the L annn ./nateria! 
on which it sets a liall-majk. 

Can it be said of English examinations at the 
present day that they Mfil these requirements ? 
Are they' wisely designed ? Do they work with 
certainty and accuracy ? Do they tend, on the 
whole, to improve the boys girls, the men 
and women, whose education, whose lives and 
careers they so largely control ? Are they really 
serving in the best possible way the national and 
imperial purposes they are supposed to serve ? 
If these questions w‘^re put to a Referendum, I 
have little doubt of the reply. 

While it may be ^conceded that there ai*e many 
judicious ^examiners, and examining bodies of 
unquestioned wisdom, the body educational as a 
whole is suffering from the aches and pains of 
examination. Those who say, Get rid of 
examinations altogether,'' offer us nothing in 
their place ; they do not realise the situation. 
I spoke just now as if examinations affected us 
©nly from without ; but have not examinations 
become, as it were, Sfi artificial nervous system 
of our education, of •which every movement is 

3 
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controlled either by their stimulus or by their 
power of inhibition, so that extirpation is an 
impossible task ? If we cannot remove this tyrant 
that governs education from within, can- 
not we either reduce its powers or force it into 
beneficence ? Quixotic as it may seem, I believe 
tfee task not to be impossible. 

The^tjsk deitiands inquiry — and by inquiry I 
mean scientific investigation — patience, and per- 
haps some drastic reform later. My object to- 
night is to suggest, first^of all, the outlines of an 
inquiry into our examination system, its design, 
its methods, and its results ; and I shall use the 
word examination '' to mean a systematic teSt 
of knowledge, and of either special or general 
capacity or fitness, carried out under the authority 
of some public body. 

First of all, the design of examinations. We 
Ccinnot understand that design without one word 
as to their history and their original function. 
The ori^n of Western examinations — I say nothing 
of those Chinese examinations which date back 
three or four thousand years — is sufficiently well 
known to the historians ; they are derived from 
the examinations of the mediaeval universities. 
Those universities were corporations of teachers 
and students, analogous to the trade and craft 
gilds of the time. In the gilds there were appren- 
tices, companions, and masters. No one was 
admitted to mastership until iie had served his 
apprenticeship, nor, as a* rule, until he had 
4 
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shown that he could accomplish a piece of woik 
to the satisfactioi! of Ih*^ gild. The obj< ct of the 
itniversity wa^ to teach ; its scholars, bache'urs, 
or pupil teachers, and mastei'^ or doctors corre- 
sponded roughi}^ to the tfiT-ee elassrs establisl’jed 
b}' the gilds, and tl>e raa'^te’* or doctor (two1:erm5 
at eniiivalont) was one ,vho. having serv^ 
his apprenticeship and passed a dc.firr’te j:echnical 
test, had received the lic^ntla docenui—the per- 
mission to teach. The candidate was required 
to teach in pmblic ahthc/examination precisely as" 
he would be required to teach later in the schools. 
The examination ^ullilled exacily the technical 
purpose for which it was inter ded. We are apt 
to think of the schoolmen and all their ways with 
a contempt inherited from the days of Bacon and 
Descartes ; but in discussing examinations of this 
kind, that distinguished historian of education, the 
late Professor Paulsen, of Berlin, asserts that they 
were well adapted to increase a student's alert- 
ness, his power of comprehending new icJbas, and 
his ability quickly and surely to assimilate them 
to his own ; and that they did more to enable 
students to grasp subjects than the mute and 
solitary reviewing and cramming for our modem 
examinations can possibly do.^ The limitations 

i Germany has frequently in the past been held up to 
English reformers as a model country in which no examina- 
tions similar to those of Great Britain are held. The writers 
have in mind cliiefly tue question of University examinations, 
and are unaware of the elaborate system of State examina- 
tions which guard every* professional career in Germany 

5 
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of the test were the limitations of the educational 
and philosophic ideals of the time. At their best, 
as I say, they did exactly what they were require*d 
to do. At their worst, says Canon Rashdall, they 
were less of a farce than certain English Univer- 
sity pass examinations almost within the mfflnory 
o£ persons now living. ‘ 

To trace th«w history of Western examinations 
ftom their origin to our own time we should need 
to follow up the ever-changing course of educa- 
tional ideals with the ^development of modem 
societies through seven centuries, and the manifold 
actions and reactions during ^11 that period be- 
tween education ideals and examination ideab. 
For such a task we have not even the materials 
prepared ; but, remembering that the original 
examination was a test of technical efficiency, 
excellently suited to its purpose, we can see how 
€*xaminations have taken on two, and perhaps 
three, entirely new functions, and with far-reach- 
ing effeocts both on examinations and on education. 

The first new function — the lise of examinations 
to distinguish between candidates of different 
merit — was an early and almost inevitable de velop- 


except that of a University teacher, and are equally unaware 
of the severe system of school examinations, condemned by 
Paulsen, and dealt with by Prof. Burnet in his book on 
Higher Education and the War. See F. Paulsen, The German 
Universities, trans. by F. Thilly and W. Elwang, p. 25, and 
Geschichie des gelehrten Untcrrichts, val. ii., pp. 368 et seq* 
^ The Universities of Europe in the Middle Ages, by Hast- 
ings Rashdall, vol. i., p, 460 (Clarendon Press, 1895). 

6 



NATIONAL EFFICIENCY 

ment ; and yet we can see every day how th^ 
competitive idea leads us to forget the distinclion 
between the efficiency of an examination and its 
difficulty. It ‘s constantly said by ex.imination- 
enthusiasts: ''This examination is a superior 
examination to that, because it iS Inuder to pass/' 
Should we not rather ask : '‘Is the'maftwho 
passes r more useful member cf society ? 1[n 

examinations prohissedly tecbiucal, ^dike the 
original examinations, this question is of the 
utmost importance. Here the main question is, 
surely, not which candidate can do the most* 
difficult things, solve the most Chinese of puzzles, 
remember the most unremeinbcrable of formulas, 
but whether the examination helping to turn 
out the best teachers, doctors, engineers, indus- 
trial botanists, industrial chemists. In other 
examinations, like the highest Civil Service ex- 
aminations, the clement of pure difficulty, on 
which I shall have a word to say later, may 
appear in another light. 

The second ne^y function of exaiiiinalions is a 
later development, and in some ways a character- 
istically, though not exclusively, English develop- 
ment — I mean their use to test not so much the 
efficiency of candidates as that of their schools. 
The external exaznination of schools and of school- 
children, begun by the College of Preceptors in 
1853, and now carried out on an immense scale 
by the College and by other bodies, mainly Uni- 
versity bodies, all over the United Kingdom, had 
• 7 
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primarily for its object to distinguish between the 
sheep and the goats of the private schools. The 
examination, as thus used, was organised pri- 
marily as a technical test, good or bad, not of the 
taught, but of the teacher. But are not the real 
interests of the taught, which it was originally in- 
tended to serve, sometimes, or perhaps often, lost 
sifht o^ in the process ? Here, again, the difficulty 
of an examination is apt to be confused with its 
efficiency. 

There is a third function of examinations, not 
*easy to distinguish altogoLher from the original 
function of testing efficiency — I mean their use 
to test what is called '' general culture '' In deal-, 
ing with this question we find a certain amount 
of common agreement as to what is the best 
policy, combined with' a powerlessness to carry 
out that policy in the face of our present organisa- 
tion of examination and education, and our 
refusal in education to remember the wisdom of 
leaving some things the time and opportunity to 

shift for themselves."' 

It will, I think, be generally admitted tliat 
culture is something more than the knowledge of 
the best that has been thought and said in the 
world, according to the definition of Arnold, unless 
knowledge is used with a far wider connotation 
than is possible in the examination-room — some- 
thing perhaps more personal and sensitive than 
the '' enlargement " or '' illuminative reason " 
wffiich John Henry Newman contrasts with know- 
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ledge.^ I shall entei into no rivalr3>^ of epigr:^i:n 
with these great raasttTs of style : but I think, on 
cross-questioning, we should find chat the cult 'acd 
man is ttie who is sensitive lespcnsive 
over a large field of knowledge legarded of 
value b};' th*'- On furtjher ^crose- 

questioning we should again hnd it generaj^ly 
admitted that that sensitiveress nnd responsive- 
ness are apt to sutler injury fn ni the iiftellectUt^l 
repression required for the examination syllabus. 
At ever^^ step the delicate feelers of the mind are. 
paralysed by the .uiggestion : I am wasting my 
time in going further; lhai won’t be asked.’' 
Jt may be held, and I should agree, that culture 
is as individual a thing as conscience ; that culture 
may be killed, that it cannot fee. caught, by 
examinations. Yet teachers who realise all this, 
who think examinations in their subject mis- 
chievous rather than helpful, implore the authon- 
ties to include it in every possible examination 
syllabus. Why ? Because, under theb present 
regime, a subject tliat is not examined in is likely 
to disappear speedily from our teaching curricula. 
That point is sufficiently illustrated by the con- 
troversy on examinations in Greek. 

We have thus seen that examinations are nowa- 
days conducted with four distinct primary 
objects : — 

(i) To test the efficiency of a person for the 
practice of his teclmical calling or profession. 

^ See Note C, p. 93, below. 
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(2) To arrange candidates in an order of merit. 

(3) To test the efficiency of schools. 

(4) To promote general culture, an object for 
which many persons regard them as totally un- 
suited. 

* I am quite aware of the complexities oLthe 
issues raise*d by this simple classification. A recent 
writer, *dcaling^ with the same problem from a 
different f)oint of view, has made out a list of 
seventeen qualities which an examination can 
^test, and thinks his list mifht be extended still 
further. My classification' deals only with groups 
of such qualities, and the grouping will vary with 
the specific object of an exariiination. I think 
it quite possible that I should find it difficult 
to conciliate all my critics, and most of all those 
who are partial to the examinations in Class (4) 
of my list. 1 am however willing, as I said 
before, to concede that any particular examina- 
tion, and. any particular examining body, may 
have all the virtues claimed for them ; but I 
would ask of those concerned jf it is not a fact 
that if we took the examination-papers set at 
many examinations, and inspected them in detail, 
we should find it an impossible task to decide, from 
internal evidence, with what object, or with what 
combination of objects, those examinatigns are 
organised. Individual examiners are not asked 
to bear in mind, nor is it, perhaps, always desired 
that they should bear in mind, the careers which* 
the examinees propose to follow ; they are asked, 
10 
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not to examine with any particular aim in vicWl 
but to examine ' ' in their subject/* and this, I 
suppose, is what is meant by the opponen;-. of 
examinations when they rail agains*. examina- 
tion for examinatior/s sake. Tb’i;^ 1 am ioc|ined 
tf> ihink that the most ardent advocate of the 
examine lion system would be prepared to admit 
that the object of many cxainmatjons held nowa- 
days is indeeterminate. It ca i scaredy be e: 
pected, therefore, that the results will be in all 
cases precise. ^ 

Before we can estirakto in detail the results of 
examinations we must say something of their 
xncthods. Examination-tests may be classified in 
two w^avs : first, according to what may be called 
their external character, under which heading we 
may divide them, as usual, into written examma- 
tions, oral examinations, and practical examina- 
tions ; and, secondly, according to a character 
much less obvious, but more important. They 
may be divided into : — 

(i) Knowledge* tests (or memory- tests),* which 
test the* power of re-stating facts and arguments 
of a kind that may be learnt by rote — e.g. ** De- 
scribe Joule's experiments on the mechanical 
equivalent of heat " ; Relate the chief events 
in themcign of Edward I/' ; '' Write out the 47th 
proposition of the first book of Euclid " ; How 
is the preposition in used in Latin ? " 

{2) What may be called capacity-tests, which 

1 Cf. p. 52 and NoteF, p.*i36, below. 
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test the power of doing something, of applying 
knowledge to some definite purpose, e.g. the power 
of making a precis of a written document, of 
writing a letter or a report on a particular subject 
with a particular object in view, of translating 
ffom or into a foreign language, or of conversing 
in ^hat language, of solving a mathematical prob- 
lem, of •fcriticisilig a passage from a literary work, 
oT writing an essay on an historical or literary 
subject with the aid of books in a library, of 
diagnosing the malady of ^ patient, of analysing 
a chemical mixture or compound ; and (perhaps, 
in certain ways, the highest form under the rubric) 
of making an original contribution to learning oi^ 
science as the result of personal investigation or 
experiment. 

In some cases the category to which a question 
belongs will depend on the previous preparation of 
the candidate. If a candidate who is asked to de- 
scribe the foreign policy of Queen Elizabeth has 
had access only to text-books in which Elizabeth's 
foreign policy is treated incidentally in a more or 
less chronological record of events, he may show 
considerable capacity in separating out in his mind 
the threads of foreign policy from the rest, and in 
putting the results of his thinking into clear lan- 
guage ; if he has been in the hands of a teacher 
who has dwelt on the particular question, his reply 
may show memory or knowledge " and nothing 
more. In subjects like political hiistory or history 
of literature, it appears to me extraordinarily diffi- 
12 
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cult to set questions in papers of the ordin<iry 
three-hours type which shall come cleedy under 
the heading of capacity- tests ^ ; wl ereas in many 
subjects the questions may be either knowledge- 
tests or capacit57“tests, and mos^ written cxanlina- 
tion-papers ioelnJ^ que- tions of both 'typ0s. 

?'he classification which I iiave here set down, 
hinted at by Mark Paitison, ana ro ^oubt Iv.- 
earlier writers, but perhap s hist churly enunciated 
in a remarkable but too little-known w^ork by 
Latham on ‘'The Action of Examiaatiens/' pub-* 
lished in 1877, obviously docs no( correspond to a 
logical dichotomy. The capacity-test of the kind 
^istd in examinations does imply knowledge, 
though the knowledge-test does not imply capacity 
beyond that of memorising, >and an elementary (and 
at times doubtful) mastery of the mother- tongue. 
It is sometimes said that the memory- tests set at 
examinations are too hard ; that the knowledge 
laboriously pumped iiuo the brains of candidates 
under examination -pressure escapes rapidly when 
that pressure has been released ; and that if we 
were to apply the witty definition of education as 
what remains when we have forgotten what we 
have learnt, we should find the result of examina- 
tion-education to be nil. It w^ould not be wise to 

Ip 

1 Mr. M. W. Kealinge, in his Studies in History (A. & C. 
Black, 1910, chap, viii.), makes interesting proposals to meet 
these difficulties. Sir Napier Shaw, F.R.S., in an interesting 
* pamphlet privately printed, suggests that our curriculum 
is largely determined by the form of the ordinary three-hours’ 
paper. , 


X3 



EXAMINATIONS AND 

accept that conclusion without reserve. It is, at 
any rate, true that examiners do not by any means 
demand that candidates should know everything 
that they are asked ; the« ordinary examiner at a 
p^ss examination is content if the examinee^can 
answer satisfactorily from one-third to one-half of 
thi qi^istions^set. When these questions are 
’ii;\emory-tests, the indulgence is surely justified in 
most cases, although there are perhaps ignorances 
of fact of a dangerous character to which this 
leniency should not apply An examiner once 
told me that on the ground that a certain candi- 
date had obtained the necessary number of marks 
in physiology (and I shall come to the question of 
marks presently), he had been allowed to pass, 
although he firmly believed that there was air* 
in the left ventricle of the human heart. The 
examination, I may say, was not a University ex- 
amination. On the other hand, for some exami- 
ners printing might never have been invented, and 
books of reference might be as, rare as they were 
in the Middle Ages. I shall not be accused of 
exaggeration when I say that a knowledge of 
technical terms in foreign languages is sometimes 
demanded by examiners which would certainly 
not be expected of candidates in their own 
language. 

In dealing with capacity-tests, the question of 
leniency and severity takes on another aspect. A * 
boy who can do right only five addition sums out 
of ten cannot add. A person who reads a ther- 
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mometer accurately five times atid inaccurately 
five times cannot read a tbermometei. A 
who understands rdne-tcnths of tte words in an 
easy passage in a foreign ianguap^e, with or with- 
outihe use of a dictionaiy, but t' ai sea in regard 
te the meaning as a wLo*c, has not brought 
his knowledge of the language to ^ useful poiSt. 
On the Ollier hand, if a candidate cm satis/p 
capacity-tests, if we are ,abie to assert from the 
definite evidence oi the examination-room, by his 
obtaining not 30 or 5oytut, say, So or 90 per cent., 
of the maximum marks assigned +0 such tests/ 
that he can do this kind of problem in mathe- 
matics, that he can speak this foreign language 
so as to make himself understood on everyday 
topics, that he can analyse a leading article in a 
daily paper so as to present its gist fairly and 
clearly to a person who has not read it, we nejd 
not fear that these capacities will fade away a 
week or a fortnight after the examing.tion. I 
firmly believe that#exairinations can tell us some- 
thing definite about candidates. The question is, 
do they at the present moment ? It is obvious 
that the answer will not be the same in regard to 
all examinations — that no general answer can be 
given to my question. But we shall understand 
its precise bearing if we deal a little more in detail 

• ‘ Cf. the evidence given by Prof. Gabriel Lippmann, the 
distinguished Professor of Physics at the Sor bonne, at the 
Enquits si^r V Enseignement Secondaire, conducted by the 
Chamber of Deputies in 1899, Procis-uerbaux, vol. ii„ p. 35. 
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with the question of numerical marks, adopted, 
not in all examinations, but in most pass ex- 
aminations and in most competitive exami- 
nations. 

The system is familiar to every one. A cejtain 
number of marks is assigned to each question set, 
afid tlj,e maximum obtainable for a paper is the 
^um of these marks. The total number of marks 
allotted to each candidate is the sum of the marks 
assigned to his several answers to those questions ; 
and the ordinary workii^^^ hypothesis adopted is 
that the value of the performance of the candidate 
is proportional to the numbei of marks he thus 
obtains. Putting the thing in another way, we 
may say that the hj^pothesis amounts to this, 
that, if we arrange candidates in the order of their 
performance as determined by the aggregate of 
marks obtained, we shall be doing them no injus- 
tice ; and this order is commonly called an order 
of merit (though order of proficiency would per- 
haps be a more accurate expression). 

The value of the hypothesis may be^ tested in 
two ways. We may examine the postulates on 
which it rests, and the results to which it leads. 
Take, first of all, the postulates. The hypothesis 
implies that we have some reasonable basis for 
determining the relative values of a series of ques- 
tions, i.e. that a number of different examiners, 
regarded as equally competent, will attribute 
approximately the same relative values to the 
individual questions ; and it implies, again, tliat 
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the same set of examiners ..will attribute app 3 t)xi- 
mately the same values ib the answers to thole 
questions given by a particular candidate. Now,, 
although examinations are carried out on such a' 
gigantic scale, we have very few puhlislit^d data 
to t 5 st the validity of these postulates, jhi ar 
statistical investigation by Professor EdgewortJj, 
it appeared that there might be v#ry gre^t dis- 
crepancies in the marks assigxied by *di£Eerq(lt^ 
examiners to the answers of a set of candidates.' 
If this is so, how can the same standard be main- 
tained ? The result is^what we might expect in 
the case of examin^atlons of which the object is 
iedeterminate, for how can we estimate the value 
of anything unless we know what is its precise 
object ? If a candidate is allowed to pass by 
obtaining from 30 to 50 per cent, of marks on 
memory-tests and capacity-tests mixed in variable 
proportions, what does a pass '' mean ? Wha^t 
may we say that a man who just passes an ex- 
amination can do ? Is not the meanimg of a 
'' pass '' in itself ifideteiminate ? Does not the 
standard necessarily vary from candidate to candi- 
date as well as from year to year ? 

If we turn again to the results of examinations 
carried on on a large scale, there are indications 
that the . results of this working hypothesis are 
not more satisfactory than its postulates. Ex- 
aminers themselves are often heard to say, ** This 
man is better than his marks, this man is worse 
^ See Note E, p. 118, below. 
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than his marks/’ ^ although perfect freedom* may 
be left to them to assign marks so as to correspond 
to resultant merit. And if we inspect published 
statistics of examinations for which a large num- 
ber of candidates present themselves, and in which 
nt wf>uld^ be reasonable to expect that ap^oxi- 
xpately the same number would attain a pass 
•standard in ^consecutive years, they show that 
there mSy be the most extraordinary variations 
in the percentage of passes in the same subject. 
Those variations, as I think any one acquainted 
with the inner working of uur examination system 
will agree with me in saying, cannot be regarded 
as due to the carelessness of individual examiners; 
for, in the first place, examiners take a great deal 
of pains with their work, and, in the second place, 
carelessness may tell as much in favour of a 
particular candidate as against him. I do not 
think that it is at present possible, without further 
scientific investigation on a considerable scale, to 
state what are the precise facts of the case, or the 
precise causes of these variations. They may be 
due to defects in our system of marking ; they 
may be due to real variations in the quality of 
candidates, of which the causes are unknown and 
which it would be important to discover. 

I want to strengthen my plea for investigation 
by quoting a witness whom I might have expected 
to find against me. If there is one subject in 
which accurate marking and community of 

^ C/. p. 6o, below. 
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standard would appear to be most easily obtain* 
able, it is mathematics ^ and it seem^ probable 
that the whole system of raarldng tv rvmbei.-^ is 
derived from the marking, of mathematical papers. 
Nevertheless, Pro%sor Eryan. a Jistinguisbed 
mathematiciaii and a hi^h authority on' th^ub- 
ject, regards some of the troubles lo which I bat^ 
referred as due to “ fetish-woibhio ' tjie maxi- 
mum mark, and suggests that we should assign 
certain marks to correct answers of average 
quality, and then mark up to any maximum we 
please, lo reward exceptional proficiency shown 
on any part o! a paper. In this way, he thinks, 
candidates would no longer feel he inducement to 

scramble for marks by scribbling scrappy at- 
tempts.’' Other authorities have suggested differ- 
ent ways of dealing with these difliculties, but it 
is safe to say that there is no agreement, and 
that different examiners employ very different 
methods. The a, 7, 8 , or Oxford system is 
another form of numerical marking in which the 
total number of marks allotted is low,* and in 
which, if \ am rightly informed, the examiners 
as a rule attempt to obtain their resultant im- 
pression of a paper as a whole directly, instead of 

by the addition of marks assigned to individual 
« 

i Articles on " Examinations," by Prof. G. H. Bryan, 
Sc.D., F.R.S., published in the School World for May and 
Jpne, 1910. 

3 By adding the signs + or — to the Greek letters, the 
number of symbols used is, as a rule, inareased to ten or 
more. C/. p. 62, below. 
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questions. Here, again, it would require investi- 
gation to decide whether this system is or is not 
more trustworthy than the other. In any case, 
in determining a man’s pass or class, you do not 
escape entirely from the difficulties involved in 
the addition of marks. Moreover, the CJreek 
letter system appears, so far as I can see, to be 
inapplicable to a competitive examination on a 
large scale in which candidates present themselves 
in a number of subjects.^ 

The assignment of marks in practical examina- 
tions is probably an easier Vnatter than in written, 
for the reason that most practical examinations, 
though not all, are capacity-tests rather than 
memory-tests, and it is easier to say whether a 
candidate can or capnot do what is required of 
him. 

Of all kinds of examinations, that which may 
be the most valuable, and that which presents 
the most difficulties, is the viva-voce examination. 
Largelymsed abroad, it is used comparatively little 
in this country, because our boys and girls are not 
taught to speak and to think at the Jame time 
at the age when children speak confidently and 
easily ; and hence, in the presence of the examiner, 
the candidate is often paralysed by nervousness, 
for which it is difficult, or impossible, to make the 
proper allowance. He may be reduced to silence 
or nonsense. A candidate in Materia Medica in 
a northern University, who waS unable, owing to 
i Sec however p. 62, below. 
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examination fright, to distinguish the taste or mz 
smell of the sample of cod-liver oil tliac he held 
iii his hand, on being asked, " Wl ere does cud- 
liver oil come from ? " jeplied, The whale/’ 

Nevertheless, the viva-voce examination has* the 
iranicnse af!vaaUigi of rdiowiiig a rea:3unabie' ex- 
ami’^er to ask a candidate not paralysed by nerws 
what porlloii of a subject he has spb/ iH^^ studied, 
and to cross-examine him considerately in that. 
The negative evidence that a candidate says 
nothing in regard to a partiraiiar printed question ‘ 
does not carry one very far. Thu frank ** I don’t 
know ” of a c.andidate, his pow^ei of distinguishing 
\ 'hat he does know from what be does not, will, 
with an examiner capable of judging men, count 
very greatly in favour of a candidate whose know- 
ledge of the parts of the subjects that he thinks 
he knows is sound. But a viva-voce examination, 
if it is to be valuable, must not be hurried, arid 
hence it is an expensive form of examination. It 
has again the defects of its qualities. The idiosyn- 
crasies of examiner and candidate may come into 
conflict in regard to matters which have nothing 
to do with the particular examination-test. To 
combat this defect, it is generally recognised that 
viva-toce examinations should, if practicable, be 
conducted by a board of examiners rather than by 
a single examiner. 

We have in the viva-voce exammation a sense of 
reality that is apt to become somewhat thin in the 
written examination. If an examiner knows that 
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he has to decide, face to face with a candidate, 
not whether he attains a particular standard in a 
particular subject, but whether he has the know- 
ledge and capacity to do a particular job or pursue 
a particular career, he will feel much more able to 
Con>%. to decision than after he has merely teen 
% candidate’s written papers. Would any sen- 
sible {jerson,«in business, in war, or in adminis- 
tration, tree to choose his own subordinates in 
carrying out an important piece of work, dream 
of doing so without an interview, if he could help 
it ? On the whole, it seems to me that our ex- 
amination system would be m(jre efficient if larger 
use were made of the oral method than is made a/, 
present. But there are administrative difficulties 
in conducting examinations by that method not 
to be imder-estimated. 

Before leaving the question of examination 
n&ethods to inquire further into the effects of 
examinations on the candidates, I want to throw 
out one suggestion. I believe that we should 
effect a quiet revolution in pass^examinations, but 
a revolution, if we would systematically ask our- 
selves in regard to each examination, What does 
it prove that all successful candidates can do ? 
What are they certainly fit to do ? Such a ques- 
tion would put capacity-tests and memory-tests, 
I believe, in their right places, and the examina- 
tion would become once more, as in its origin^ 
what the mathematicians call a directed quantity. 

And now for the effect of examinations on candi- 
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dates. If we take a large cold ihermometer. as 
accurate an instrument as please, ard plnnge 
if into a relatively small quantity cf warm w^.ter, 
the reading ot the thcrpiemf ter \vA] not tell us 
directly what the temperature ci the waternvas 
before we trircl nicasnre it. In the samei^aj^’; 
ever the most admirably designed and organi^d 
examination will generally exercis« a very con- 
siderable disturbing influence on the** previous 
education of the candiiate that it is designed to 
test, and on the candidate himself during the - 
examination. In the 'case of the thermometer, 
we can ceiTect our original r<^adhig by making the 
necessary allowance ; in the c?se of the examina- 
tion candidate, what correction can we possibly 
make ? 

How, if we cannot calculate them precisely, 
shall we reduce the bad influences and magnify 
the good influences of examinations as far ws 
possible ? It is a grave problem, and we shall 
find considerable* disagreement in determining 
what features of e'^aminations belong to one cate- 
gory, anfl what to the other. Let us, for the 
moment, consider the influence of the examina- 
tion on the candidate while it is being actually 
carried on. There can be no doubt that examina- 
tions prolonged over several days, for from four 
to six hours a day or more, involve a considerable 
physical strain on the candidates, and the sample 
of knowledge and capacity which the examina- 
tion affords can hardly be regarded as an average 
23 
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sample. With some candidates the excitement 
and strain make them do better than they other- 
wise would do ; with others the reverse is the case. 
Ought we to try to reduce that strain ? A good 
many, I fear, would not assent. It has been 
said^and pot infrequently said, that if a candidate 
fa^s under the strain, he is lacking in some of 
those (^ualitio# which go to make a successful man, 
and that though failure in the particular examina- 
tion does not mean (as the public might, and often 
. do, think) that he has not the intellectual attain- 
ments of those who pass, yet that no harm is 
done. It seems probable — cannot go further 
than that — that failure in examinations owing ta 
temporary ill-health takes place more frequently 
in the case of women candidates than men, and 
the question of women candidates certainly needs 
separate investigation ; but in the absence of 
adequate statistical and medical inquiry, the whole 
hygiene of the examination-room remains obscure. 
Much more important, in the long run, than the 
immediate influence of the exjfmination-room on 
the candidate is the influence of his preparation 
for the examination during the long years of 
school and university life. That is a subject too 
complex and important for me to do more than 
mention ; I would only suggest two things — first, 
that in some subjects and with some candidates, 
the influence of examinations may be, and often ^ 
is, good and not bad ; and secondly, that instead 
of encouraging by means of examination subjects 
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ill*suited for examination-tests, we might tr) to 

protect them from those tests. I do not that 
it is easy, but I think it not impossible. 

Let me take the subject of liislv/ry, which, oi 
all subjects that come under the heading of 
culture ” subject:^ rather bread-and-bwtter 
subjects, is tlic luosi iniportarit to the ordir^ry 
citizen. Would it not be possible to reduce 
school examinations in ory to a ^minimum 
by allowing those who had attended a history 
class ill a secondary school, to the satisfaction* 
of a duly qualihed teacher, to be exempted froui 
any kind of schopl-leaving examination in his- 
tory? Of course, it will be said that the stan- 
dards of different teachers would be different, but 
are we not inclined to sacrifice on the altar of 
equality the very’ efficiency of the teaching that 
examinations are intended to test ? I remem- 
ber, in a French Lyc6e, hearing a lecture on Napo- 
leon's coup d'etat of the i8th of Brumaire given 
to a class of science boys who were not to be 
examined in the subject. The lecture was full of 
warmth ^d colour, every boy was interested, 
every boy was listening ; and 1 contrast that 
with the painful dictation of slow sentences, 
reproducible in the examination-room, that I 
have heard in the history class-rooms of some 
English secondary schools. 1 do not mean that 
all French teaching of history is good, and that 
ail English teaching of history is indifferent. I 
believe there is no subject in which more rapid 
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progress is being made in this country. Might 
not the examining bodies help the history teachers 
in other and better ways than by inserting his- 
tory in all their syllabuses, as the teachers seem 
to desire ? ^ 

* InMealtng with this matter, I would suggest 
thai it is as impossible to make certain that every 
boy or girl ^fho leaves school has an adequate 
knowledge of history, has a real beginning of 
historical culture, as it would be to make sure that 
"every boy or girl who leaves school has musical 
culture or artistic culture*. Supposing you do 
make sure by examinations . that the school- 
leaving certificate implies a certain knowledge of 
the facts of history, what then ? A boy coming 
out of the examination-room once said to me : ** I 
am never quite sure whether the date of Magna 
Carta was 1215 and Waterloo 1815, or whether 
it "was the other way about, but I think I have 
got it right this time.'* A boy of that kind, even 
with the more difficult questions set nowadays, 
might absorb history, but he would never assimi- 
late it, and instead of trying to force htm to be 
cultured in a subject where culture is impossible 
for him, would it not be much better to recognise 
his insensitiveness on the historical side, and push 
him in some other direction ? What I have said 
of history applies even more strongly to literature. 
If we could only make up our minds to require, 
real efficiency in one or two subjefets, and give our 
boys and girls the opportunity of learning other 
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subjects, without forcing them mechanically; to 
do so, would not our education becon^ much more 
effective in the end ? Are we not led by the wiU- 

0 “the~wisp of geiieraU culture ihto a mire df 
general inefficiency ? I asked a friend of# mine 
who is an experienced examiner, what it «^ould 
be safe to assert of the powers of a student who 
could just succeed in obtaining pa-^s marics at a 
pass examination in several subjects. His reply 
was : "I think you could say that he could pass 
the examination at the time that he passed it.'"* 

1 went further. '' Would it be safe to say that 
he had any useful knowledge of the subjects of 
the examination — that he would be fit for any 
career ? '' The answer was clear : " No, you 
could say none of these things. When you have 
said that he has passed the examination, you 
have said all that can be said/' 

If we turn from pass to competitive examiiia" 
tions, we find that they involve all the same prob- 
lems, and fresh ones to boot. Here it is well 
recognised that difficulty may be used as the 
means, not of testing proficiency in subjects that 
will be required in later life, but to distinguish 
different candidates in regard to what is called 
their '' general ability." • The great choice of 
subjectl> permitted by the regulations for the 
Open Competitive Examination for the Home and 
.Indian Civil Services sufficiently illustrates this 
point. That particular examination is one of the 
greatest importance. Is it really fulfilling its pur- 
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pose in the best way possible ? Professor E. G, 
Browne, of Cambridge, in giving evidence before 
Lord Reay's Committee on Oriental Studies 
(Minutes of Evidence, page 62), said that the 
men selected by the method devised by Lord 
Croiffer, in connection with the Egyptian Services, 
we§e more apt in the learning of Oriental 
languages thSh the men selected by open com- 
petition for the Indian Civil Service.^ I believe 
Lord Cromer intends to say something, later, 
•on the methods that he used. I do not say that 
Professor Browne's isolatefl observation proves 
anything conclusive, but it ^’s significant, and 
points to inquiry. 

The tremendous severity of the Civil Service 
examinations has bet;n regarded as imposing a 
sufficient test of the character and of the physical 
strength required for arduous public service. But 
dd we know at the present moment whether the 
game is worth the candle ? A letter from a 
candidate, who wrote over the somewhat unfor- 
tunate signature Presque passe," in the Times 
of last summer, reminds one uncomfortably of 
the sad picture of overworked Indian examina- 
tion students drawn about the same time by Mr. 

[now Sir] Valentine Chlrol in his articles on Indian 

* 

1 Minutes of evidence of Treasury Committee on the 
Organisation of Oriental Studies in London (Cd. 4561), 1909, 
Questions 1740-50. Prof. Browne attributed the difference* 
to the prominence given to mathematics and science in the 
competitive examination. 
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Unrest.^ Is it not possible to test '^.general 
ability/' and to separate \he ablest candidates Jbjr" 
methods involving less strain both on the siicceks- 
ful and the unsuccessful candidates ? Tjie ex- 
p^imental psychologists and statisticians, work- 
ing on lines suggested by the late Sir Fjflncis, 
GaHon, hold out hopes in this direction, butj^ne 
must confess that they are stid som^hat remote. 
Would it be possible, without reintroducing the 
unspeakable evils of jobbery, to follow the lines 
laid down by Lord Cromer in the Egyptian Civil* 
Service, and by Lord*Selborne in choosing candi- 
dates for the Navy ? These are questions again 
'for inquiry.* 

1 Reprinted as a volume, Indian Unrest, 1910, see pp. 217- 
18. 

* Referring to examinations for the Indian Civil Service, 
Army, and f-Jome Civil Service Clerkships of the Second 
Order, Prof. Edgeworth writes [Journal of the Royal Stafis- 
tical Society, voi. liii., p. 653) : — " I find the element of 
chance in these public examinations to be such, that only 
a fraction — from a third to two-thirds — of the successful 
candidates can be regarded as quite safe, above the danger 
of coming t)ut unsuccessful if a different set of equally com- 
petent examiners had happened to be appointed." With- 
out being an expert in statistics, I venture to think that the 
basis of Prof. Edgeworth's calculations is too slender, and 
now too much out of date, for|us to draw general conclu- 
sions from them ; the uncertainty of the examinations in 
question may be to-day much greater or much less than 
that shown by those calculations. The serious matter is 
that we are entirely in the dark as to the actual situation. 
Xl reprint this footnote as being an integral part of the 
original paper. For a fuller account of Prof. Edgeworth’s 
work, see Note E, p, 102, below. — P. J. H.] 
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Conclusion 

How should such inquiry be carried out ? I 
do not think that any other body than a Royal 
Gonwiiissijn would carry with it the necessary 
weight. I believe the Commission should be a 
small d!ie, pr^ided over by a statesman with great 
experience of affairs, and that there should be no 
attempt to achieve the impossible by including 
in it representatives of all parties concerned. If, 
following the example set« in Professor Sadler's 
Moral Education inquiry,' and in the recent report 
of the Consultative Committee of the Board of. 
Education on Evening Schools, the evidence 
taken were published, not in the unreadable and 
unwieldy form of question and answer, but in 
the fonn of statements on particular topics sum- 
marised by the witnesses from shorthand notes, 
after their examination by the Commission, every 
person and every interest would have a fair say ; 
and the members of the Commission themselves 
would not have any undue advantage m giving 
their own summary of the evidence, and their 
own conclusions. But such a Commission, to do 
all that it should do, ffould need to be far more 
than a Commission of Liquiry in the ordinary 
sense ; it would need to be a Commission of In- 
vestigation. It should have the power to appoint 

t 

' Moral Imtfuciion Training in Schools: an Internaiional 
Inquiry, ed. by M. £. Sadler (Longmans), 1908. 
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and* to pay committees of experts in different 
subjects, who would carry ojat investigation under 
its direction. Take the questions of stari4ar(| 
and of marking. Those^questions it is difficult td 
investigate adequately without having a fcirge 
niJhiber of candidates' answers lithographed of 
printed and sent to different examiners to^be 
marked, first without any directions at all, and 
secondly with directions relating to th? stress to 
be laid on particular points regarded as of especial 
importance. Nor would the expert in his subject * 
alone be necessary ; the assistance of the statis- 
tician would have, to be called in at each step. 

I believe that in this way we might be able to 
make an examination standard that would mean 
something very different frojn what it means now, 
that would be reasonably, though not rigidly, 
constant from year to year, and that would sub- 
stitute the fairness of considered judgment fdr 
the unfair impartiality of chance, in the case of 
far more candidates near the borderline. That 
would be one of the matters for the Commission 
to investfgate. If it were to appoint sub-com- 
mittees in a half-dozen, or a dozen, subjects of 
examination, the results of the inquiry would 
probably throw sufficient tight on the subjects 
which it* was not possible to touch. 

A second question for investigation is the 
hygiene of the examination-room. How far are 
we making fair diemands on our students in the 
examinations ? How far do they exceed, from 
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the point of view of strain, what it is reasonable 
to demand ? 

A third point for inquiry, and that the most 
difficult, is the influence, of examinations, good 
and bad, on education as a whole. I would only 
^ggest here that different classes of candidaTes 
anjJ different subjects require different treat- 
ment, ^nd tlmt it should be the object of the Com- 
mission tt) find some way of protecting from 
examination those school subjects which are un- 
• suited to them, and in which examinations dig 
up the roots of the plants which we wish to grow. 

Fourthly, the Commission yvould inquire how 
far our different sets of competitive examinations 
select the persons most fitted for the posts which 
they are intended to occupy. 

In this connection, the Commission would, no 
doubt, consider the question of associating as 
eKaminers with the experts in their particular 
subjects administrators with experience of the 
services for which candidates were being chosen. 

I have selected four large topics for inquiry : 
(i) the question of standards and marRing, with 
which is closely connected the question of the 
precise object of the examination concenied ; (2) 
the hygiene and psychology of examination^ ; (3) 
the influences of examinations, good and*bad, on 
previous training ; (4) competitive examinations 
for the Services. But these topics would sub- 
divide, and others would suggest themselves for 
investigation. A Commission with a suitable 
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reference would find tio ^ack of useful work to do. 
I suggest as a reference : “ To investigate and 
report upon the methods and efficiency for taeir 
purpose of examinations canicd on Govern- 
ment Departments a id other paHHc bodies ifi tlie 
United Kingdom ; to inquire into the mfluehce'> 
of examinations on the previous educatiorypof 
candidates ; and to suggest sueh chaTiges as may 
seem desirable/' 

The investigations or such a Commission would 
be of immense assistance both to the teaching and 
to the examining bodfts of this country. I have 
tried to obtain figures showing the magnitude of 
our examination operations. Those figures are 
still far from complete, but I have official statistics, 
supplied to me by the authorities concerned, which 
show that in the year ending September 30, 1910, 
out of 146,741 candidates at public examinations 
in the United Kingdom, 71,114 failed. I haVfe 
little doubt that when my figures arc complete — 
and I hope I may be permitted to publish them 
later — the total entries will approach 200,000 and 
the failures 100,000.^ Does there not seem to be 
something very wrong in the relations of our 
examinations with our education for nearly half 
the persons to be judged unfit to pass the test for 
which they present themselves ? How far is this 
great percentage of failures due to a low average 

* 1 Several bodies w^dch were good enough to furnish me 
with statistics made it a condition that the details should 
not be published, 

D 
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of intelligence or industry, how far to the fault 
of the teachers, or the fault of the examiners, or 
to want of co-ordination between teacher and 
examiner? Conscientiously conducted in their 
details, have not our examinations been largely 
at the nTercy of hazard in regard to principles ? 

f or^ Rosebery, speaking in 1907 as Chancellor 
of the University of London,^ urged strongly that 
examiners needed training no less than teachers. 
He also suggested the need for investigation. 

* For such teaching as Lord Rosebery desires I 
doubt if any common basi^ of doctrine at present 
exists. It must be created. < 

There is one serious obstacle to change. The 
examiner and the organiser of examinations are 
often so weary of their ungrateful tasks that for 
them the day’s work is enough ; they become 
hardened, sceptical of improvement, reconciled to 
eVils that they have come to regard as unavoid- 
able evils. But the issues are too grave to be 
overlooked, and the investigation and reform that 
I have ventured to demand are a matter of 
national necessity. 

Speech by the Late Earl of Cromer, 
O.M., G.C.B. 

Lord Cromer, speaking from the Chair,* said : — 
I am sure that you will all agree with me when I 
say that we owe a deep debt of gratitude to Mr. 

» At the opening of the London Day Training College ; see 
the Tim$s of November 4th, 1907. 
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Harlog for the very interesting pap)er to v/liich 
we have just listened. It L a paper whicli stimu- 
lates thought, and, let me add, the^e are few sub- 
jects more deserving of thought at present than 
that to which Mr. Hartc*g lias drawn our atten- 
tion. Wo seem h) be about to enter a period* 
during which, whether for govd or evil, the 
either in its imperial or municipal vapa‘Cii^ is 
likely to develop gnat acti/ity in tlfe perfor- 
mance of a variety of new functions. How are 
the large additional numbei of agents whose ' 
services will thus be required to be recruited ? 
The results which ^^vill be obtained by the new 
departure now being made will, hi a great measure, 
depend upon the answer which is given to this 
question. It is perhaps an^ exaggeration to say 
with Pope that '' Whate er is best administered 
is best,'' but it is certainly true that a good 
system may be marred by mal-administratioiv, 
whilst the defects of a faulty system may be 
greatly mitigated if the administration is efficient. 

There are, so far hs I know, only three methods 
under wflich public functionaries can be ap- 
pointed. These arc election, selection or nomina- 
tion — the two words are, for the purposes of my 
present argument, synonympus — and competition, 
either open or restricted, 

I need say but little of the elective method. Its 
advantages are mainly political rather than 
administrative. An eminent French economist, 
M. Leroy Beaulieu, has said : According to 
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democratic theory, popular election will always 
secure the most capable individuals ; but this is 
a curiously mistaken idea, and is quite unsup- 
ported by experience/' 

The Poor Law Commission, over which Lqjd 
Deorge Hamilton recently presided, took the same 
vi^v. The method of popular election," the 
Commissioiters reported, " is not one which is 
adapted Vo get the best administration. The 
Commission believe that there is a better chance 
' of good administration if the members are 
appointed than if it is left*to the chance of elec- 
tion." I will not, however, t dwell any further 
on this branch of the subject, as it touches the 
fringe of political issues, which I am anxious to 
avoid. I will only,, therefore, say in general 
terms that one, and perhaps not the least impor- 
tant of the questions which the democracy of 
this country will have to face in the near future, 
is how to combine administrative efficiency with 
an adequate amount of popular control. I have 
often, in the course of my Eastern experience, had 
to deal with a very similar problem, anh I gener- 
ally found that the question practically resolved 
itself into this — How much efficiency is it 
justifiable to sacrifice ^on the altar of self-govern- 
ment ? I came to the conclusion, urider the 
special circumstances with which I had to deal, 
that it was impossible to generalise on this sub.- 
ject, and that each separate issue, as it arose, had 
to be decided on its own merits. 
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« 

Taming to the other alternatives, viz., selecucn 
or competition, it may readily be admitted that 
competition, which is the prevaiiii g system now, 
has its ch^feetb. but before condemning H, it is 
worth while to glance back at the system which 
ii supplanted. T ,mi one of the band, not now 
very numerous, who can speak with some per^j^pal 
experience of the pre-competition dkvs, anS, as 
illustrations of the working of the tinfli existing 
system, I may perhap^ be pardoned if I relate a 
few biographical details. Some fifty-six years* 
ago, being then a boy oi fourteen, I was admitted 
into the Ro37al Military Academy at Woolwich, 
with a view to eventually bee ming an officer in 
the Artillery or Engineers. At that time there 
was a sort of sham entrance-examination, but I 
never heard of any boy who had been nominated 
by the Master-General of the Ordnance being 
rejected. Lord Hardingc, the grandfather of 
the present Vicero}^ of India, who was a friend of 
my famity, gave me my nomination. On pre- 
senting myself at Woolwich for medical examina- 
tion, I wS,s very rightly rejected for short sight. 

I returned to London and told my mother, 
who was my only surviving parent. She acted 
with promptitude. She instantly rang the bell, 
ordered her carriage, and went to the Horse 
Guards to see Lord Raglan, who had succeeded 
to Lord Hardinge's place, and who was another 
friend of my family. The result was that next 
day I returned to Woolwich with a letter addressed 
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by Lord Raglan to the medical officer, asking him 
to reconsider the matter/' I was, of course, 
admitted . Exactly the same thing happened at the 
same time to another lad who was at first rejected 
on the ground that he had a serious impedimojp.t 
in his speech, but whose case was subsequently, 
un4j4,^i^ressure, “ reconsidered/' This sort of 
thing couI3 not happen now. The practical 
working of the system may, however, best be 
illustrated by an anecdote which is related of 
' Lord Melbourne. A friend of his who occupied 
a high position and who disposed of a good deal 
of patronage, said to him : “ I do not in the least 
mind confessing that if 1 had to deal with two 
candidates, one of whom was the son of a friend 
or relation of mine, and the other a stranger, I 
should, ccicris paribus, give the appointment to 
the son of my friend or relation." To which 
Cord Melbourne drily replied, " So should I, but 

ceteris paribus be d d." In fact, at that time, 

the principle laid dowm by George III had only 
been but slightly modified.* That monarch is 
believed to have said that any one was fit to occupy 
any place he could manage to get. ' 

Well, ladies and gentlemen, as a result, partly 
of the attention called to military matters by 
reason of the administrative breakdown during 
the Crimean War, and partly of the extraordinary 
call for officers occasioned by the Indian Mutiny, 
the nomination system, in so faf as appointments 
to the Artillery and Engineers was concerned, 
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was swept away, and the competitive system iri^ro-' 
duced. It started under rather bad auspices. 
I remember that the first notice issued by thi War 
Office was rather unfortunately ’/oided. They 
called for " persons ” to compete. The result 
was that all the young men who responded to 
this call were instantly dubbed “ persons,” and 
commanding officers showed a marked 
for those who had entered unuer the £)ld system, 
and who had enjoyed the more exalted title of 
” gentleman cadets." I was at that time an 
ardent young Liberal, as I claim to be an old 
Liberal now — although my right to that desig- 
nation may perhaps not be recognised by mere 
party politicians — and I was warmly in favour of 
reform. As a matter of fact, the gloomy pre- 
dictions of the anti-reforfhers were falsified by 
the event. The social class from which the officers 
who entered the Army w'ere drawn was just the 
same after as it had been before the change of 
systems, whilst there can be no doubt that a dis- 
tinct improvement was effected, both in intel- 
lectual standard and professional capacity. 

I mention these points mainly in order to show 
that the competitive system, whatever may be its 
defects, is greatly superior to that which it super- 
seded, and in order to warn the advocates of 
reform that they must beware lest a door be 
opened which may lead to a revival, perhaps in 
a somewhat different form, of the abuses of the 
past. 
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And now let me make a few remarks on the 
competitive system considered on its own merits. 
I think the best thing I can do is to deal with facts 
which have come within my personal experience ; 
and, as you are probably aware, my personal 
Wperience has been gained more in the outlym|* 
portions of the Empire than at home. I have 
''rrrrminfr^^ irnl of the work performed by the 
young mei\ who are turned out by our universities 
and military colleges. I have met them on the 
J^anks of the Ganges, the Indus, the Irrawaddy, 
and the Nile. And I have also had some rather 
exceptional opportunities of comparing their 
aptitudes with those displayed by their Conti- 
nental rivals and competitors. From one point 
of view the comparison is disappointing. I have 
often had it said to mb by heads of departments 
in Egypt, themselves Englishmen, that the young 
F^^nchman or German came to them crammed 
to the muzzle with an amount of preliminary 
knowledge greater than that possessed by the 
young Englisliman. The explanation is, I think, 
that until recent years sufficient attentioR has not 
been paid in this country to technical education ; 
that is unquestionably a defect, more especially 
at a time when there is a tendency to. specialise 
every branch of leamirig, but it is a defect which 
I hope and believe is in process of being remedied. 
But if we look at education not merely as a means 
for storing the mind with knowlejdge, but also as 
an instrument for developing the character, the 
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situations are reversed. The youth in whom the 
need of individual effort and the senst; of personal 
responsibility have been nurtured in one ox our 
public scJioolb or colleges, becomes almost uncon- 
sciously capax imperii. He is no automaton* 
In the free atmosphere in which his boyhood and 
youth are passed, h€ learns a number of lessons 
which stand him in good sb*ad in cfi'. 4iW*as a 
member of an imperial race. xVlore ( specially, he 
learns not only to Ihinh, but, which is much more 
important, to a('t for himself — in a word, to* 
govern, and to govern generally with firmness, 
justice, and wisdofn. From this point of view 
our system of making appointments, which is 
generally, though not exclusively, the result of 
competition, may, I think, in spite of some defects, 
be said generally to have' produced satisfactory 
results. A very high authority on Indian affairs, 
Mr. Valentine Chirol, recently wrote : "'To the 
Indian civilian belongs the credit of almost every 
measure passed during the last fifty years for the 
benefit of the Indian masses, and passed fre- 
quently iii the teeth of vehement opposition from 
the Indian politician.'* I think you will agree 
with me that this is very high praise. The system 
has thus turned out a number of young men who, 
as the agents of a nation whose very existence 
depends upon the execution of a sane imperial 
policy, are not only of incomparable merit, but, 
I do not hesitate to add, are also the envy and 
admiration of the whole civilised world. What- 
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ever changes may in the future be made in our 
educational system, I most earnestly trust that 
no steps will be taken which may tend, in 
any degree, to weaken that sense of independ- 
ence in thought and action which should com 
Stitute the basis of character to the individual^ 
who are the units of any self-governing com- 

In spites however, of this success, I think we 
may well ask ourselves, after the experience of 
•half a century of the working of the competitive 
system, w^hether it is not capable of improvement. 
The question is certainly worthy of consideration. 
Remember that the principle of selection, if only 
it can be properly carried out, possesses merits 
superior to those of competition. The former 
may, or ought, to result in the creation of leaders 
of men. It should enable us to acquire for tJie 
public service the aristocracy, in the proper and 
original sense of the term ; that is to say, not the 
aristocracy of birth, but that of intellect and of 
character. The latter tends lather to produce a 
dull level of mediocrity. Most educated English- 
men have recently been reading the history, 
written by a very distinguished contemporary 
statesman, of one of his most illustrious predeces- 
sors — I mean Lord Rosebery's history of the 
early life of Chatham. Now, what was it that 
enabled Lord Chatham to revive British spirit 
when it had sunk to a very low^bb, to march to 
victory on the banks of the St. Lawrence and the 
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Ganges, and to found the British Empire • I 
say that it was the adoption of the principle of 
selection, sparingly, indeed, but Vi ry skilfully em- 
ployed. I notice with pleasure that of late years 
there has been a tendency, notably in the military, 
naval, and diplomatic services, to adopt this 
principle in dealing with all the later stages of the 
careers of public servants more thc'.o'ugriJJ' than 
was formerly the case. I trust that^^'this move- 
ment, far from being* arrested, will be pushed 
still further. I could name numberless instancesf 
especially in the |i:iitary service, in which the 
dead weight of^^id regulations, based generally 
on the claims of seniority, have acted detrimen- 
tally to the public interests. 

The case of first appoinj:ments presents natur- 
ally far greater difficulties, for, ex hypothesis there 
can in these cases be no record of past services 
on which to rely, and moreover, here, and 
indeed in any application of the principle of selec- 
tion, we are at once met with the awkward 
question, Quis cuUodiet ipsos custodes ? Who is 
to select* the selectors ? 

In connection with this subject, there is one 
point in respect of which I cannot help thinking 
that a good deal of misapprehension exists. It is 
not uUfrequcntly thought that Ministers and 
others in whose hands the bestowal of patronage 
is vested, would rather welcome an accession to 
their powers in the matter of making first appoint- 
ments. I believe this view to be wholly erroneous. 

43 



EXAMINATIONS AND 

My experience has led me to the conclusion that 
those who exercise patronage are also those who 
most of all delight in being fettered by regulations 
which relieve them of a very irksome and onerous 
duty, in the exercise of which they are certaii]^ 
tb incur ^ great deal of misrepresentation and 
abuse, and to receive very little gratitude. It is, 
in facfT that the wider is the discretion left 
to a Minisli^r the more fully will he be exposed 
to the full blast of social, and also — which is 
perhaps even more hurtful — of parliamentary 
pressure, to make his sel^tions on grounds 
other than those of merit. Mpivover, unless he 
be an exceptionally strong man, it is well-nigh 
certain that he will at times succumb to this 
pressure. More than this, it has to be remem- 
bered that neither the action of Parliament itself, 
in its collective capacity, nor the power exercised 
by* the Press, however honest in intention, affords 
any adequate guarantees either against deliberate 
jobbing, or what is perhaps a greater danger, 
against unintentional negligerfce. The truth is 
that, save in a very few exceptional cases, the 
number of persons who can speak with impar- 
tiality and full knowledge of the merits of a candi- 
date, whether for prornotion, and still more for 
first appointment, is very limited, and, metreover, 
it is not unfrequently difficult to obtain a frank , 
expression of their opinions. I do not suppose, , 
therefore, that any one, however fully he may 
recognise the defects of the competitive system, 
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would be prepared to advocate a return to tbe 
principle of uncontrolled selection. 

Let me again fall back on my personal expefL 
ence, and explain how, in one special instance, to 
which Mr. Hartog has alluded, 1 solved the diffi- 
culty myself. Some few years ago it became 
necessary to create a Soudanese Civil Service. 
In the first instance the appointniLnts** were 
practically made by myself. I fuun€ it a very 
difficult task. I used to receive hundreds of 
letters from the friends or relations of candidates.* 
Of these about 90 ^" xr cent, found their way into 
the waste-papetf®' gasket. I took an enormous 
deal of trouble to select the beot men. Indeed, I 
may remark incidentally that whatever success 
has attended the administration of Egypt during 
the last thirty years has been mainly due to the 
great care which was taken both in selecting and 
in promoting officials. Generally speaking, the 
method adopted in the first instance in the 
Soudan for making first appointments worked 
fairly well. Nevertheless, I felt that, apart from 
the greaf trouble and responsibility it imposed on 
myself, the system, or perhaps I should rather 
say, the absence of system, was very unsatis- 
factory. In the first plape, I never could feel 
sure th^t the merits of many young men perhaps 
,more deserving than those who were actually 
Appointed had not been overlooked. In the 
second place, although I felt quite certain that I 
should not job, I felt almost equally certain that, 
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sooner or later, I should be accused of jobbing, 
and the accusation would in itself have done 
harm, inasmuch as it would have shaken public 
confidence. On the other hand, I was not pre- 
pared to resort to open competition, as I was, 
convinced that many of the qualities most re- 
quired for a Soudanese civil servant could not 
be te?[edrT)y a competitive examination. Even- 
tually a system was adopted under which the duty 
of selecting candidates, who are all taken from 
*the universities, was, under conditions which 
I need not describe in detailT^;onferred on a board 
of highly qualified officials, were left quite 
free to act as they thought best in the public 
interests. The system produced good results, 
but I do not at all say that on that account it is 
suitable for general adoption in this country. 
There are alternatives on which I need not dwell. 
yty present object is merely to draw attention 
to the subject, and to indicate the advisability 
of considering whether it is not possible to adopt 
measures which will on the 6ne hand admit of 
some independent control being exercised over 
State authorities, and more especially local 
authorities, in cases where the competitive 
system, properly so qallcd, is not applied, and 
which will, on the other hand, tend to Yemedy, 
or at all events to mitigate, the acknowledged 
defects in the competitive system itself. T^*" 
test supplied by that system, though in some 
respects valuable, is purely intellectual, and 
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indeed, as at present applied, it may be doubted, 
more especially after the evidence adduced by 
Mr. Hartog, whether, even as an intellectual 
test, it always succeeds in attaining its object. 

I Time does not permit me to dwell on the details ; 

I will therefore only say that, in laying down the^ 
conditions under which a competitive examina- 
tion is to be conducted, the most important 
points to be considered are, first, h#w are the 
examiners to be appointed, and, secondly, on 
what principles should the}^ proceed in awarding * 
marks. : 

For the rest, ^ should like to point out that 
the Order in Council of 1870 <xntemplates the 
possibility of a departure from strict rules in 
certain cases, but such a departure can only be 
made with the consent of the Civil Service Com- 
missioners. It is in the development of this idea 
that, for my own part, I should be inclined to 
look for reform. In any case, I hold a strong 
opinion that if anything is to be done in this 
direction, Parliamdatary politicians should be 
rigorously excluded from participation in any 
Boards or Commissions which may have the 
disposal of patronage. 

Ladies and gentlemen, \ fear I have detained 
you too long. I trust, however, that this discus- 
sion will lead to the matter on which Mr. Hartog 
j|ias dwelt being considered by others more com- 
petent than myself to deal with them. 

The idea of appointing a Royal Commission to 
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examine into the whole subject is certainly one 
which commends itself to me. 

— The paper and speech printed above, together 
with the discussion which followed, were published in the 
Journal of the Royal Society of Arts for February 3rd am 
loth, 1911; In the course of the discussion the proposals 
for an investigation of the examination-system were sup- 
portedgby Sir Henry Miers, F.R.S., Professor John Adams, 
and Professor Arthur Schuster, F.R.^. ; they were also 
warmly suppaortcd by Professor M. E. Sadler in a letter 
printed in the Journal for February 3rd. I have made a 
, few slight corrections in the text of the paper, and added one 
or two footnotes.] 
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THE THEORY OF EXAMINATIONS 

L DEriNITION 

1. For the purposes of this article the word 

" examination " (cxcv*pt where otherv/ise indicated) 
is used to mean * a* systematic test of knowledge, 
or of either special or general capacity or fitness, 
carried out under the authority of some public 
body.’' The argument, foia the sake of brevity, 
mainly refers to examinations as carried out in 
the United Kingdom. ^ 

II. Origin and Objects of Examinations 

2 . Our modern examinations are based on 
mediae val'^examinations designed with a view to 
testing whether a candidate had the necessary 
qualifications to practise a craft or profession. 
They have since been applied to the following 
new purposes : (i) to arrange candidates in order 
of merit (competitive examinations) ; (2) to test 
proficiency in a subject, or in one or more sub- 
jects, and, following on this, to test “general 
culture ; (3) to test the efficiency of schools. 
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8. The distinction between professional ex- 
aminations on the one hand, in which the exa- 
miners are guided by a definite and definable 
(though often complex and unanalysed) object in 
view, and non-professional examinations on the^ 
bther, in which there is no such object in view, 
serving as a guide, is of a fundamental character. 
Most* of the anomalies of our examinations are 
found in ^on-professional examinations, or in 
those parts of professional examinations based on 
• the model of non-professional examinations. 

4. Intennediate between professional and non- 
professional examinations art examinations of 
which the purpose is definable, but of which the 
definition of purpose conveys widely differing 
ideas to different minds. The chief examina- 
tions in this category are (i) " School-leaving 
examinations,” (ii) professional entrance examina- 
tbns, and (iii) University entrance examinations 
(Matriculation Examinations). The first are 
supposed to show evidence of a good general 
education, and to imply fitness to enter a 
business or professional career, withotit specific 
limitations ; the second, to imply fitness to enter 
on a course of study for a particular professional 
career (e,g, that of doctor or solicitor) ; the third 
to imply fitness to pursue a University career. 
Frequently one and the same examination is 
used for all three purposes. 

5. The examination system has acquired an 
immense importance in all civilised States, and 
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has become an essential factor^ in the social 
organism, since the selection of individuals to 
fulfil various functions m the State, as we^' as 
for the great categories of social occupation, 
largely determined by examina cions. It has been 
calcuJated {Journal of Royal Society of Arts, 1911, 
p, 1132) that there weje in the United Kingdom 
in 19x0 o\cr 300,000 examination candidati^ in 
elementary and secondary subjects only, apart 
from University and professional exalRiinations, 
and the relative numbers of candidates in the 
United States, France, Gennany and other large 
countries are probably not less. 

6 . Nevertheless, Ifherc are few questions relat- 
ing to examinations which can be regarded as 
uncontroversial, owing probably to two reasons : 
(i) the uncertainty of social and educational 
ideals at this time of unrest (even before the 
War); and (2) the fact that examination testj 
have never been submitted to any scientific 
investigation, and that we are largely in the dark 
as to their validity fpr the purpose for which they 
are employed, and sometimes even as to the pur- 
pose itself of a given examination and the mean- 
ing of the certificates conferred in connection 
therewith. 

III. Classification of Examinations 

' 7. The kinds of test used in examinations may 
be classified for different purposes in a number 
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of ways (which necessarily overlap to some 
extent), as follows : 

{a) According to the method by which answers 
to the examination-questions are furnished by 
the candidates: (i) written, (ii) viva-voce, (iii)^ 
.practical (including clinical). Examinations may 
combine these tests in various ways. 

{by According to the period occupied by the 
examination work, which may (i) be relatively 
short and Timited by the examining body, extend- 
ing over a period of from a few hours to a few 
days, or (ii) extend over a long period, generally 
without any maximum limit. * Examinations 
of course-^^'ork done under *the supervision of 
teachers, and of theses and dissertations, come 
into this second category. 

{c) According as they are conducted (i) by per- 
sons not appointed to examine as teachers, i,e, ex- 
t^iirnal examiners — it is to be noted that in some, 
though not in all, external examinations the 
examiners may and do include persons who have 
taught some but not all of v the candidates — or 
(ii) by teachers of the candidates, or«(iii) by a 
combination of categories (i) and (ii), a frequent 
combination in University examinations in the 
United Kingdom. 

(d) According as they are (i) knowledge (or 
memory) tests, or (ii) capacity-tests, or (iii) 
a combination of the two. A knowledge-test 
shows the power to reproduce in a more or less 
modified form information acquired by the can- 
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didate ; a capacity-tes+ shows the power to do 
something other than to reprodacc, e.^. to 
a foreign language, to read a thf i niometer, to 
auscultate a patient. 

{e) There are also psychological tests, con-# 
ducted by expeiinv^u^^al iiiothods. 

(/) Tests of medical fitness. 

IV. CritetUA of Tests 

8 . The word examon means the pointer of a 
balance ; the analogy of tne testing of the 
physical properties' of a material substance with 
the testing of the fitness and capacity of a human 
being is obvious, and the criteria of physical tests 
can and ought, with certain important limita- 
tions and extensions, to be applied to examina- 
tions of human beings. The principal criteria 
for physical tests may be stated as follows : 

(a) The test used of a given characteristic 
should demonstrably yield a suitable measure of 
that characteristic. This is not a self-evident 
proposition (thus it is practically impossible to 
measure directly such a property as “ absolute 
temperature," as defined in thermo-dynamics), 
and it is necessary to show how far indirect 
measurements are valid for their purpose (see 
under (ii), § 9, below). Many examination 
methods are indirect tests of the characteristics 
which it is desired to measure and compare. 

(b) In cases where the whole of the material 
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under investigation cannot be tested, it must be 
shown that the sample tested is an average 
sample of the whole. 

{c) The test must be trustworthy,'' i.e. it^ 
► must, when repeated a number of times as nearly 
as possible under identical conditions, by the 
same observer, or when repeated by different 
observers,^ yield approximately the same results 
(the competence of the observers being assumed). 

(d) The application of the test itself must not 
sensibly affect the characteristic tested, or in- 
juriously affect the material as a whole. 

9, For the sake of brevitj^ we shall, in what 
follows, use the word '' characteristic," as applied 
to examinations, in a wider sense than usual, to 
denote anything for which marks may be given 
to a candidate, e.g, knowledge of French, or of 
mathematics, capacity to ride a horse, etc. The 
numerical evaluation of human characteristics 
presents at least two kinds of difficulty which 
exist only to a much less extent with physical 
characteristics, ^ 

(i) Whereas wc can in many (though not all) 
cases legitimately represent the values of physical 
characteristics either in terms of a unit, by 
straight lines of different lengths, or by numerical 
ratios, such representation in the case of human 
characteristics, though convenient, is often of 
doubtful validity. There is n9 such thing as 
" unit knowledge of mathematics," or " unit 
knowledge of French," nor could we justify a 
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statement that candidate A knows twice as 
much French or mathematics as candidate B/* 
We shall deal further wi^Ti this matter below. 

(ii) The variation of human characteristics both 
with lapse of time and with t}ie conditions under 
which human btii^gs are normally placed is much 
grcn.ter, and less easily determinable than the 
variation of the physical char: cteristfts ol 
material substances in daily use concurrently 
with the variations Of the physical conditions 
under which they are usually placed. Without 
going into an elaborate analysis, we may say that, 
while it is useful , for practical purposes to say 
that the density of a piece )f iron is 7*8, by 
which we mean that it differs little from that 
number under every-day conditions of tempera- 
ture and pressure, it is more difficult to make a 
corresponding rough-and-ready statement in 
regard to the important mental characteristics, 
either general, such as ''power of attention,’* or 
specific, such as " pow’^er to give an account of 
the Norman Conquest,’* since these characteristics 
vary under every-day conditions of interest and 
fatigue, and the law of such variations escapes 
definition. 

10. It is clear that the j^nowledge of a charac- 
teristic of a purely evanescent kind is of little 
value. A candidate who knows the dates of the 
accession and death of the English sovereigns 
merely as a series of numbers on one day may 
have forgotten them on the next. Unless, therc- 
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fore, there is collateral evidence of the likelihood 
of the permatience of the knowledge itself, the 
only permanent or semi-permanent characteristic 
of which such knowledge shows evidence is the 
power to memorise numbers, and if we are ignorant 
to the time taken to me morise them the char- 
acteristic is one to which we are not entitled to 
attach any great value on the evidence shown. 

We may ^d to the criteria of examination tests 
enmnerated above the following : (e) the char- 
•acteristic tested should be demonstrably of a 
permanent or semi-permanent nature. 

It frequently happens that, while examiners 
nominally assign marks for knowledge, they are 
perfectly aware that the knowledge may be 
evanescent, and they really intend to assign 
marks for what is assumed to be the pennanent 
power of acquiring knowledge. 

•We shall use some or all of the above criteria 
in dealing with a few of the more important prob- 
lems which arise in connection with examinations. 

V. The Problem of Purpose 
\ 

11 . Before dealing with more special problems, 
we have to deal with the general problem of pur- 
pose, the most important, but, in very many 
cases, the most neglected of all examination 
problems. The majority of discussions about 
particular examinations, other than final profes- 
sional examinations, are confused because the 
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object and purpose of these examinatiom* Is not 
defined, and in the cas^ of ' pass /nd * hop^^nrs 
examinations because the examine’-^ do not te!i 
us what a person who just passes '' or ]u<yt gets 
third ” or fourth-class honours/' knows or can 
do. From expern ncc we know that we cannot 
say with any certainty liiat a person who 
passes " In a given subioct will even be able 
to pass in that subject a year later. ^Probably if 
the examiners were pressed they w^ould admit 
that '' passing " the examination means nothing 
more than that a certain candidate was examined 
by certain examiners ana was able to earn a 
certain percentage of marks Li a certain number 
of subjects at a certain date. We have therefore 
next to investigate the problem of marks and 
marking, 

VI. The Problems of Marking 

12 . Most examinations are made up of a series 
of tests, either of the same or different kinds. A 
number of marks is assigned to each question in 
each paper, or viva-voce, or practical examination 
set, and the maximum obtainable is the sum of 
the marks so assigned.' The total number of 
marks allotted to each candidate is the sum of 
the marks assigned to his several answers to the 
questions which he is asked to answer. The 

^ If a candidate is required to answer only a certain frac- 
tion of the whole number of questions, the maximum is 
reduced accordingly. 
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ordinary wor|j:ing hypothesis adopted is that the 
value of the performance of the candidate is 
proportional to the number of marks he obtains, 
and that we shall be doing no injustice to the 
.candidate as a whole, if we arrange them in an 
** order^of merit (better expressed as order of 
proficiency) in accordance with the numerical 
sums of such marks. In pass examinations a 
candidate i§ ordinarily required to obtain from 
^ 30 to 50 per cent, of the grand maximum assigned 
to all the tests which he is required to take, in 
order to be allowed to '' pass.'* 

To satisfy criterion (c) § 8 above, we should be 
able to show (i) that at a given examination a 
number of examiners, regarded as equally com- 
petent, will assign approximately the same relative 
mark-values to the different questions; (ii) that 
t|j[e same, or another set of equally competent, 
examiners will attribute the same marks to the 
several answers to those questions given by any 
candidate.* Further, if we ^ are to be able to 
compare the proficiency of candidates of any year 
with the proficiency of those of a preceding year, 
we must have some adequate comparison of the 

standard " at the two examinations in question, 
an exceedingly difiicuU and complex problem, of 
which nothing but a rough-and-ready solution 

' And that the same examiner wiU allot the same marks 
if he marks the papers a second time. In examinations with 
hundreds or even thousands of scripts tc^ mark, the question 
of the examiner’s fatigue becomes of importance. (Sec 
Note E, p. Ill, below.) 
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can be provided. It has been proposed in 
exammations at which large nun^bers of 'Candi- 
dates present themselves to ens.'re a constant 
standard of '' passing ” by allowing the same per- 
centage to pass each year, thus assuming the 
variations ol marks obtained to be due 'to varia- 
tions in the standard of the difficulty of the 
questions and of the maikm*£ of tiie examiners, 
or both. ^ 

13 . The published evidence in regard to con? 
stancy of marking, notwithstanding its funda- 
mental importance, 's exceedingly slender. Edge- 
worth in 1888-90, in an investigation on First 
Class Civil Service Examinations, found the 
element of chance in the marking to be so great 
that only a fraction of from a third to two-thirds 
of the successful candidates could be regarded 
as above the danger of coming out unsuccesfrful 
if a different set of equally competent examiners 
had been appointed. ‘ The element of chance 
may be expected a prion to vary wdth the purpose 
of the examination and to be least in a final '' 
professional examination, where, in awarding 
marks, the examiner has more or less consciously 
in his mind the relative importance of the tests 
in relation to the requircTOents of the profession 
and the fitness of each candidate to enter the 
profession. Where, as in a pass examination in 
" arts '' or science,'' the purpose of the exa- 
mination is not defined, the meaning of a pass 
^ Sec Note E, p. 1^9, below. 
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is necessarily indeterminate. Corresponding to 
this indetermination we find remarkable varia- 
tions in the percentage of candidates who pass 
from year to year. It may be safely assumed that 
these variations are not due either to variation 
in the pjercentage of marks required for a pass, 
or to carelessness on the part of the examiners. 
Any variation in the percentage of marks required 
for a pass is probably made with a view to reducing 
rather than to increasing differences in the per- 
centages of passes '' from year to year ; and 
carelessness may easil}/ tell in favour of as well 
as against any particular candidate.^ Without 
careful statistical in vestigai ion on a large scale 
it is impossible to state what are the precise facts 
of the case or the precise causes of the variations 
of passes. Tliey may be complex, and due [a) to 
real variations in the average proficiency of the 
candidates, or (6) to variations in the methods 
of setting papers and of marking the answers by 
the examiners, or (c) to a combination of these 
causes. 

14 . We have assumed, so far, that the total 
marks earned by a candidate are the sum of the 
marks earned on the different parts of the examina- 
tion, but it is shown by experience that, exami- 
ners are frequently not satisfied that the totals of 
marks assigned in the way described represent 

^ In well organised examinations speci'^l precautions are 
taken to eliminate error through marks being wrongly added 
or overlooked. 
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the proficiency of the candidates, and are apt 
to say '' the candidate is a good ifiark -getter, or 
a bad mark-getter as the case may be/’ ^ Pro- 
fessor Bryan, F.R.S., a remedy,* ha« pro- 
posed to abolish absoJu^-e maxima and to leave 
examiners free to add on marks fer exception’* 
ally good answers. The seme result might be 
more conveniently achieved b^^ e/'signing (say 
20 -per cent.) marks for general impression, so 
that the total marks might be made to corre- 
spond to the "'resultant impression” of the* 
examiner, which is probably, though unanalysed, 
a safer guide than the mere addition of marks, 
in the existing absence of sta'^istical information 
as to the general trustworthiness of the marking 
system. Some examiners give negative marks 
for answers showing want of comprehension, but 
the practice is not general. Experience seems to 
show that examiners are more frequently in agree- 
ment as to the ” class ” of a candidate than as to 
his marks, the number of "" classes ” assigned being 
not more than four. In the minds of examiners 
there sometimes appears to be a real if undefin- 
able distinction between the classes of two candi- 
dates, A and B, w^ho have obtained the same 
numerical marks in the first instance ; they may 
say A IS a first class ” man, B is a ” second 
class ” man ; and the separation of the two may 
be finally effected by an adjustment of marks. 

1 Stanley Ldlithcs, What is Education? p. 154. 

a Loc. cit. See note, p. 19. 
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Possibly it might be shown that examiners 
in this way distinguish between evidence of 
" capacity and of " memory " or knowledge. 

15. In the * ** Greek-letter ** system of marking 
each answer is marked a, 7 , or S, in the first 
instance^ with (say) four further distinctions 

(+, + +, 0 attached to each of the first 

three letters. The system has some advantages, 
but it is difficult to see how it can be applied to 
a competitive examination on a large scale unless 
4:he symbols are translated into numbers. ‘ It has 
been suggested that the only satisfactory way of 
determining order of proficiency '' (or merit 

is to compare each candidate's‘^’performance with 
that of each of the others and to assign a final 
order in accordance with the number of superi- 
orities showm by the several candidates. The 
process is too laborious to be applicable to any 
laige number. 

16. Some of the greatest anomalies in marking 
arise from the failure to distinguish the princi- 
ples applicable to {a) examinations designed to 
test the progress of a pupil, from those applicable 

* Certain examiners assign Greek letters to individual 
answers in the first instance, and then convert them into 
numbers wliich are added cp, so as to give a total for each 
individual. Thus, if the system indicated in the text is used, 

we have sixteen different symbols, to which are attached 
the successive numbers zero to 15. There is probably a 
real psychological difference between this process and what 
might seem to be the equivalent proce«s of assigning 15 
mailcs to each answer. 
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to (b) examinations designed to test practical 
capacity. Take itie case simpleiaddition, and 
suppose that the candidate can only do two 
addition sums right out of six. If I. he examiner 
is only testing the progress of Ihe candidate, it 
may be legitimate to allot to him 30 per <;ent. of 
the marks ; but, if the examination is designed 
to test his power of adding . orrectlv, he should 
receive no marks, since that power has not reached 
the point of practical u'^cfulnebS. The same kind 
of argument is applicable to all kinds of capacity 
tests ; for these there should be no mark allotted 
in examinations in category^ (b) above unless the 
candidate obtains considerably more than half 
(say 75 to 90 per cent.) of the marks assigned 
to the particular test ; but the argument does 
not apply to memory tests, in which a candidate 
is legitimately entitled to marks for each item 
remembered. • 

17 . In w^hat precedes we have preferred order 
of proficiency ** to “ order of merit " because in 
the anonymous examination, as ordinarily con- 
ducted, we know nothing of the merit,” or 
deservingness, from the moral point of view, of 
the candidate.^ Speaking roughly, proficiency, 
as shown in an examination room, may be said to 
depend inter alia on three Variables, (a) natural 
ability, (b) opportunities (including opportunities 
of getting good teaching), and (c) industry and 
perseverance. With a particular purpose in view, 

^ See Note G, page 138. 
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it may be justifiable to place A, who is a clever, 
lazy dog, abcfve B, who is less clever but more 
industrious ; with another purpose in view, the 
order ought to be reversed. In considering a 
permanent appointment to any post, it is clear 
that examination results taken by themselves 
are insufficient. 

When the results of a competitive examination 
can be madf to depend in part on the report of 
trustworthy persons familiar with the previous 
* history of the candidates some account may be 
taken of the relative importance of the three 
factors (a), (6), and (c), in contributing to the 
proficiency shown, and weight may be given to 
each factor corresponding to the general purpose 
of the competition. 

\II. Problem of the Examination-sample 
See criteria (b), (c), and (e), §§ 8 and 10 above 

18. Examinations may be roughly divided into 
two classes, according as (i) the examination itself 
covers the whole field to be tested and the results 
give complete information as to the character- 
istics required,' or (ii) the field of characteristics 
can only be tested by sample.'' We can show 
with sufficient comple'teness by examination that 
a candidate has the power accurately to add, 
divide, or multiply, to read a thermometer, or 

^ To simplify the problem, we neglecf here the variability 
of condition of the candidate (see § 22 below). 
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measiire an electrical resistance, the ;^wer to 
write in his mother-timgue dearies grammati- 
cally, and in accordance with a logical plan, to 
speak a foreign language, to virile suorthand, and 
a great many other things of primary practical 
importance.* • 

If the examination be sufficiently complex we 
can, as in the case of actuaries, covm the whole 
field* of a professional examination, ^so as to lie 
able to say with certainty, this candidate can do 
everything required of an actuary. But the ' 
majority of examinations test only by sample. 
As a typical examjDle of cases where judgment 
by sample fails, wc may quote that of an examina- 
tion in '' unseen Latin '' in which the examiner 
happens to pitch on passages with which the 
candidate is acquainted. The conclusion drawn 
in such a case is obviously wrong. Even such 
complex professional examinations as those m 
medicine cover only a small portion of the field. 

19 . The question of fairness of sample only 
approaches to deffiiiteness when the field is 
clearly defined either by the well-known require- 
ments of a profession or by examination syllabus. 
Otherwise the field itself will vary necessarily 
with the examiner, and the comparative success 


1 The fact of failure to pass such an examination does 
not necessarily prove that a candidate would not show the 
characteristic tested, if he were placed under conditions 
other than those pi the actual examination (see § 22 
below). 

F 
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of two candidates equally well equipped in every 
way will depend on whether their views (or those 
of their teachers) of what is important coincide 
or not with those of the examiners. 

20. Let us consider, first, professional examina- 
tions by sample. The professional examinations 
which the public probably regard as of the greatest 
importance are those in medicine. Here the 
field of minimum requirements, though constantly 
varying, is fairly well defined at any given time. 
In order to make sure that a candidate shall not 
succeed by chance, the public insist that he 
shall have gone through a definite course of 
instruction certified by his teachers, in addition 
to passing the examination conducted either by 
his teachers, or by “ external examiners,'' or by 
a combination of the two. In most other cases 
public opinion is content with less stringent 
guarantees to provide against the possibility of 
success by chance, and to ensure that the ex- 
amination-sample shall be a fair sample of 
capacity and knowledge. 

21. In the case of non-professional examina- 
tions by sample, e,g, honours examinations in 
history or literature, the element of chance is 
greatly increased, especially when the examiners 
have no knowledge of the previous triining of 
the candidate ; and where such examinations are 
competitive the element of chance would appear 
seriously to affect the validity of the result (c/. 

§ 13 above). 
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22 . The question how fair a particular l^andidate 
is in a normal condition during th^ examination 
is a special aspect of the problem o; sample 

(1) The candidate may, by chance, be ki an 
abnormal condition, due to temporary indis“ 
position. It iS imi^ossible in most examiflatiens 
to make any correction to allow for this ; yet 
failure in the case of a candidate who is ill at tljiC 
time 'of the examination gives no evidence as to 
his powers when he is well. 

(2) We know that capacity-characteristics are 
far more permanent than memory- character- 
istics ; tlicy imply Jhabits ox thought, not easily 
destroyed or lapsing with time. From one point 
of view memory-characteristics may be regarded 
as evidence of the capacity to memorise, but the 
ordinary examination gives no information as to 
the time which has been required to memori^^ 
in a particular case. It is only in the case of 
examinations of extreme difficulty, like the Class 
I. Civil Service examinations, with an age limit, 
and covering a veiy wide field, that we get valu- 
able evidence of the power to memorise rapidly. 

(3) The candidate may be in an abnormal 
condition due to the examination itself. Some 
candidates are stimulated by the excitement of 
examinations to a display of powers of which they 
are not normally capable ; with the majority, 
owing to nervousness and fatigue, the reverse is 
probably true. 

The question of fatigue is one which demands 
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investigation. Does, for instance, a written ex- 
amination conducted for six hours a day on five 
successive days diminish the average mental 
powers sensibly, and, if so, does it diminish them 
in approximately the same proportion for all 
candidJites ? Do we definitely wish to eliminate 
certain candidates by this fatigue-test ? Will 
the results in subjects requiring special concen- 
tration, ^.g.cmathematics, be affected according 
as the papers in those subjects are placed at the 
beginning or the end of such an examination ? 

(4) The question of nervousness specially 
affects viva-voce and practical examinations. The 
question whether allowance should be made for 
such nervousness at a viva-vocc depends on the 
purpose of the examination. If it is definitely 
intended to test the candidate's power of ” think- 
ivg on his feet before an audience, no allow- 
ance should be made. If it is intended only as 
a test of knowledge or capacity in a special sub- 
ject which will not be tested in the same way 
in the career for which the candidate is intended, 
the test itself may be a failure, if the candidate 
is nervous.^ In many foreign examinations the 
viva-voce is definitely used as a test of power of 
exposition. The case, of the practical examina- 
tion is somewhat different, as practical tests have 

^ When the purpose of an examination is properly defined, 
it is easy to show in certain cases (as -syhen it leads to the 
selection or passing of an unsuitable candidate) that the 
examination has failed. 
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rarely to be carried out before a critical audiei ce, 
and it is expedient if pcssible so to conduct 
practical examinations that a candidate shall 
noi be able to fail entirely throujin a sinjrle act 
due to nervousness. 

23 * Teshni; of Schools by Sample — A#spec.ial 
aspect of the probieiii of sample is presented when 
the efficicnc}^ of a school is judged by testing d 
certain proportion of its pupils selected either at 
random, or belter, from a classified fist drawn up 
by the school. The testing, by an external ex- 
aminer, of a few pupils classified by the teacheib 
as the best, the average, and the worst may be 
made to yield a fair comparis'^n of his own judg- 
ment with that of the teachers. 

VIIL Influence of Examinations on 
Education and Teaching 

24 o In 1889 the Hon. Auberon Herbert pub- 
lished a collection of opinions of distinguished 
people entitled, The Sacrifice of Education to Ex- 
aminations, and Mr. Stanley Leathes has prob- 
ably expressed what is still the popular view in 
regarding examinations as a necessary evil '' 
{loc. cit,, p. 141). But, as a matter of fact, the 
influence of the examination on the candidate, 
whether for good or bad, depends entirely on 
whether it is well or badly designed for its pur- 
pose, and the general question “Are examina- 
tions ' good ' or ' bad ' ? “ is indeterminate, 
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i,e[ it cannot be answered in the negative or the 
affirmative, tif it is essential for a candidate's 
future that he shall be able to speak French to 
strangers at a given age, his education up to that 
age will not be sensibly influenced by the fact 
that he- is required to pass a viva-voce examina- 
tion in French ; similarly with the examination 
for actuaries (see supra). Here the influence of 
the examination, considered by itself, is nil. The 
nature of flie examination and the previous 
•education are alike determined by a consensus of 
opinion, and the examination qua examination 
does not influence the candidate or his teachers 
for good or for evil. The fact remains that in 
many cases there can be no such consensus of 
opinion ; and in these cases the examination may 
exert an independent effect, prejudicial or bene- 
ficial, or a combination of both, on previous 
education. 

It is in the nature of things, especially in non- 
professional examinations, that there should be 
great diversity of opinion both as to the proper 
field, and even the purpose, of a particular ex- 
amination, and that diversity will become most 
marked in those branches where progress is being 
made and recent developments appear in rapidly 
changing perspective. ‘ The purpose of to exa- 
mination being defined, it is open to question, 
in any giv^en case where there is disagreement 
whether the view of a particular jamming body 
or whether the view of a particular teacher is 
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best adapted to the ftiWlment of that mirposj.' 
But it is certain that where there h such conflict 
of view either the candidate's teaching musi be 
subc^rdinated to the views of the exaininmg body, 
or he must risk failure because he is being 

trained with one |Hirpose in viev/ and tested with 
another. If the teacher examines or has a suffi- 
cient share in the design of the examination the 
alternative of conflict or subordination does not 
arise. Where it does a:‘ise the persoifal efficiency 
of the teacher is bound to be diminished if he is 
forced to teach in a way which he regards as 
second best. Professor T. P. Nunn has pointed 
out, on the other hand, that examinations serve in 
certain cases to stimulate teachers, but those 
cases are limited to certain categories. li is 
clear that where teachers have some share in the 
examination of their students the examination 
syllabus is more flexible and can be more readily 
changed from year to year. (For question of 
anonymity and fairness of examinations conducted 
by teachers see § 29, below). 

25. Effect of Examinations on Powers of Exposi^ 
tion , — In ordinary life a sensible person only 
speaks or writes on any subject for the benefit of 
a person who knows less tlian himself ; the 

^ The independent judgment of the candidate himself, 
important in questions of culture, is only in the rarest cases 
likely to be equal or superior to that of examiners or teachers 
in other matters, and the sacrifice of the candidate's own 
judgment in trainin}^' for examinations is generally inevitable 
(see infra, § 27) . 
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examination candidate writes for a person who, in 
general, already knows what he has to say, i,e» 
the examiner, and hence may be trained un- 
wittingly to express himself obscurely and by 
allusion. It is of great practical importance to 
counteract this powerful tendency by rational 
training in composition.^ 

Sir Napier Shaw thinks that the actual form 
of the three-hours' papers now set has exercised 
an adverse Influence on the teaching given in 
’•physical science.* 

26. Effect on Powers of Concentration . — In the 
case of many candidates, examinations, through 
tb^ fear of failure, induce concentration during 
working-time which it is difficult to secure in any 
other way. 

^ Cf. The Writing of English, by the present writer, 
PP* 73- 74 3.nd passim. 

^ “ An evil microbe lias got hold of the teaching of physical 
science, for I know of my own knowledge that some at least 
of the examiners outside the examination-room take a real 
interest in things. I attribute the sawdust with which the 
examinations are stuffed to the influence of [a] prolonged 
succession of examinations of the same type, the three hours* 
written form of exercise wdiich corresponds with no problem 
that any man or woman has to face in practical life, except 
the single one of keeping awake while the examination lasts. 
Why can one not invent some other form of examination, 
one, for example, in which candidates are asked, they are 
in practical life, to take up a job and say when they have 
hnished it ? ” — The Lack of Science in Modern Education, 
by Sir Napier Shaw, Sc.D., F.R.S., Director of the Meteoro- 
logical Office (Lamley Sc. Co., 1916), pp. 13, 14. The pamphlet 
is in part a plea for the study of " real tfiings,** as opposed 
to ** laboratory science.’* 
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27. The Effect on Education of Examinations de- 
signed to test Culture , — If culture distmgiilsbed 
clearly from eruditioD, end be interpreted as au » 
individual sensitiveness and responstvor ass in 
regard to those things which can be neither 
weighed nor measuiecl hut are nevertheless re- 
garded as of value by the commiuiity/ then it 
may well be held that cuh.mc cannot be tested by 
any formal examination, but only b^' social inter- 
course, or by the evidence of original work (thesis 
or dissertation), A written examination can test* 
a knowledge of literature, or a knowledge of 
history , it cannot test the literary or the historical 
sense. But the training for s^ich examinations, 
by checking the individual impulse of the candi- 
date at every turn, may well kill the individual 
culture and originality of effort which it cannot 
test. The sensitive inquiries of the examination 
candidate are likety to be checked at every turn 
by the inevitable question — Will they ''pay '' at 
the examination ? 

28. The Effect of Examine lions on Education by 
Over-absorption of Time , — If examinations cannot 
test culture, still less can they test (except in 
certain limited ways) character, or public spirit. 
It is therefore of the first importance that the 
curricula of candidates, and especially of candi- 
dates in secondary schools, shall be designed so 
that training in the subjects tested by examina- 
tions shall only absorb a portion of the candidate's 

1 See Notes C and D, pp. 93 and 95 below. 
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time. In exceptional cases the effect of prepara- 
tion for examiiiations is to ruin a candidate's 
.health by over-work. The effect of examinations 
on the health of girls and the precautions necessary 
to avoid overstrain in their case deserve special 
considemtion. 

IX. Impartiality and Fairness of Exami- 
• nations 

• 29, As applied to examinations, the terms 
impartiality and fairness are not synonymous. 
In cases where the results of examinations are com- 
petitive, it is of the highest importance that they 
should be, as far as possible, not only impartial, 
but fair in the sense that the candidates shall be 
really arranged in order of suitability for the 
purpose for which the examination is held. 

Examinations in which the candidates are 
entirely unknown to the examiners are necessarily 
impartial, but the impartiality may be to a con- 
siderable extent that of chance, and have all the 
injustice of chance. To have anticipated, or to 
have been coached, in a number of memory-tests 
which are later selected by the examiners may 
bring success, but it affords no proof of superior 
suitability. It is to be* pointed out that anony- 
mity can only be fully preserved when an examina- 
tion is wholly written ; as soon as the candidate 
is brought face to face with the p.xaminer at a 
practical or viva-voce examination, the anonymity 
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may disappear. The more searching ana personal 
tlie test, the greater are the possi^uJitves both of 
fairness on the one hand and of paxtiality cn the 
other, everything depending on the character and 
capacity of the examinees. 

Possibly the viiM satisfactory method af selec- 
tion IS to make a first choice anonymously, by a 
written examination, and then to \crt the surviv 
ing candidates by viva-vcce examination before a 
Board, either with or \/ithout evidence from those 
previously acquainted with the candidates. Such 
a Board may of course be prejudiced against 
particular candidates and categories of candidates, 
and against such fJrejudice there is no safeguard 
but public opinion. As an example of successful 
selection by boards of examiners may be quoted 
the method used by Lord Cromer in choosing 
officers for the Soudan. ‘ The efficiency of such 
methods of selection is tested by the results. 

X, Class I. Civil Service Examinations* 

30. These competitive examinations are of a 
peculiar type. The candidates may select, with 
certain limitations, as many subjects as they like 
out of a large number (to which arc allotted 
different maxima), as long as the total number 
of marks obtainable does not exceed a certain 

1 See p. 45 above. 

» See also Introduction, p. xii, above, and Note A, 
p. 8o, bUow. 
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limit, and there is no necessary relation between 
the subjects which a candidate takes and his 
future career. It is clear that this is a case in 
which the purpose of the examination is not to test 
directly special knowledge of any kind, but to test 
the power to master a number of subjects ; the 
severity of the competition makes the prepara- 
tion for the examination to some extent a test of 
the candidate's character, inasmuch as to ensure 
success he must for a time sacrifice all other 
interests to it. Success at the examination is 
followed by different requirements according to 
the service entered by the candidate. Those 
selected for the Home Civil Se^rvice go through a 
period of probation without furthe r examination ; 
those selected for the Indian Civil and certain 
Colonial services have to undergo further pro- 
bationary examinations in special subjects re- 
quired for their official work. 

A fundamental criticism of the methods 
adopted at these examinations is that it is impos- 
sible to compare the suitability of the candidates 
by means of heterogeneous aggregates of marks 
assigned by the different examiners in different 
subjects. Since on the whole the system has 
worked well it may be assumed that the number 
of suitable candidates is considerably irf excess 
of the number selected in each year. It is com- 
monly stated that marks may be more easily 
earned " in subject A than in subject B, but the 
validity of such a statement in respect of different 
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pairs of subjects will almost necessarily van’ irom 
year to year, § 

XL Suggested Use of Examinations to test 

Common Sense and Power of Judgment 

81, Hitherto comparatively little use h«s been 
made of examinations to test common sense and 
power of judgment in a for which skil/ul 
preparation could not famish auy^ illusory sub- 
stitute. The writet has suggested that '' English 
papers may be set of a kind co test '' common* 
sense, intellectual tact, reediness ard order/' * 

«; 

XIL Experimental Psychological Tests 

32. A large amount of important work has been 
done recently by psychologists with a view to 
determining the natural capacities of individual, 
'' ediicands " b}^ experimental methods. Prof. C. 
Spearman and Dr. F. H. Hayward have sug- 
gested that experimental methods may replace or 
coalesce with the examination methods hitherto 
used. In America experimental methods have 
been successfully applied by F. W. Taylor, Miin- 
sterberg, and others to determine relative fit- 
ness for various kinds of industrial work. The 
methods seem capable of great development ; 

1 The Writing of English, pp. 81-4, and paper on Examina- 
tions, read before the British Association, published in the 
School World, October 1911 ; the new English papers pro- 
posed by Mr. Stanley Leathes* Committee for Class I. Civil 
Service Examinations should serve this purpose (see p. 92). 
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and a system of tests is being used in American 
vocational bureaus ” for the purpose of advising 
' youths in the selection of a career. It is possible 
that the whole system of examinations may be 
ultimately transformed by the use of these 
quantitative methods ; but the matter is still in 
a rudimentary stage. 

XIII. Medical Tests 

* 33. For certain careers a medical test of fitness 
i? essential. There is a movement on foot to 
extend medical tests and to give marks in Univer- 
sity degree examinations for^ physical capacity 
as shown in Swedish drill and the gymnasium, a 
return to Greek ideals. 

XIV. Conduct of Examinations 

34. To avoid chance hardships many precau- 
tions of detail are necessary in the conduct of 
examinations, of which the following are a few 
examples, taken at random as illustrations. 

(I) It is the practice of some examiners to 
attribute different mark-values to different ques- 
tions, leaving it as an additional test of the candi- 
date s intelligence to determine in what way they 
can earn the largest number of marks wdthin the 
period of the examination. Definite instructions 
on this point and on the minimum number of 
answers which must be attempted to ensure a 
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pass should be given if the examii^ing bod| 
holds a particular view on the m^atter. 

(2) In the case of mathematical papers tha 
examiners should work out the questiops i:i fujl, 
to avoid unexpected difficulties and even *' traps/* 

{3) It is essential tlial examiners should dot be 
^eq^:ired to mark an excessi ve number of scripts 
in a givcii time. (Sec p. 110/ 

« 

XV. Conclusion 

35 . In what precedes a method has been 
sketched out and illustrated, for dealing with 
examination problems. But the question is only 
in its infancy. A sj^stematic investigation (in- 
cluding statistical investigation) of the method 
and results of examinations is urgently needed. 
A great advance would be effected if examining 
bodies could be induced to express in words (i) 
the purposes of their various examinations, and 
(2) as exactly as possible what the passing of 
a particular exammation means in terms of what 
all the successful candidates could certainly do 
at the time of the examination — in other words, 
non-professional examinations should be given 
the same precise significance as professional 
examinations. 
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EXTRACTS FROM THE FOURTH REPORT OF 
THE ROYAL COMMISSION ON THE CIVIL 
SERVICE (MAJORITY REPORT), 1914.1 ' 

(i) First two paragraphs of Section 7 {p>, 29) 

7. But before proceeding to discuss in any detail 
the schemes of examination bv which we propose 
that those tests should be carrh.d out, we desire to 
emphasise our opinion that an examination scheme 
is only part, and not the most important part, of the 
machinery by which the State should aim at the selec- 
.^mn of suitable candidates for its service. 

Unless the genera] educational system of the country 
is efficient, the natural powers of those who would 
make the best candidates will neither be discovered 
nor trained. 

(2) Third paragraph of Section 8, and first two para- 
graphs of Section 9 (p. 31) 

8. In Chapter I we have indicated the origin and 
followed the development of the competitive system 
of recruitment for the Civil Service. With the pro- 
gress of this development the efficiency of the Service 

1 [Cd. 7338.] Published by the Stationery Office. Price 
IS. 4^., to be purchased of any bookseller. 

80 



EXAMINATION S— N OT E A 

has grown until at the present time the Sla*; e p^ssesGe| 
a body of public officers who are far jnore coni net etit 
and zealous than their predecessors, -tjipointed under^ 
the regime of patronage, are stated on official autho- 
rity to have been. We have no doubt whatever that 
to this highly sati^vfactory result ihe sy‘=tem com- 
petitive examinaticn has mainly contributf'd. The 
system has. in oar opinion, mtirely justified the 
expectations of its originators. It is true that it 
never has been, niid is not now, j, exempt from 
hostile criticism, and we do not affirm that the 

• 

written examination is an infalliDle or a final test 
of the best results of education. But the defects 
attaching to the system, which we shall in due place 
examine, are, perhaps, susceptfile of some mitiga- 
tion. In existing political conditions, and in such 
developments of them as can be reasonably antici- 
pated, we believe that the advantages of the system 
of competitive examination as a means of rccruitmei^ 
for the Civil Service far outweigh any defects whiim 
have come to our notice, and we arc convinced of the 
importance not only of adhering to ^he system, but 
of extending it whenever possible. 

9. Since, then, examination should, in our opinion, 
play so large a part in the selection of this most 
important body of officials, it becomes of supreme 
importance so to order and arrange the schemes and 
improve the methods of examination as to secure 
the best* results. 

How far the existing arrangements secure this will 
appear from a review of the schemes of examination 
by which at present access is gained to the different 
administrative and clerical classes. We propose 
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now to undertake such a review, and to recommend 
such changes a^ appear desirable. 

(3) Sections 26 and 27 (pp. 34, 35) 

26. We have already drawn attention to what is, 
in ou'r judgment, a serious defect in the existing 
arrangements of the examinations for entry into 
the clerical branch of the Civil Service, viz. their 
divergence from the actual conditions of education. 
The Second Division examination marks the extreme 
* point of such divergence. 

The three years seventeen to twenty correspond to 
no stage in the organisation of education in this 
country. As a complete course; of secondary educa- 
tion at present ends when the pupil is about eighteen 
years of age, it is obvious that, while the inferior age 
limit is now too low, the superior limit is far too high. 

27. An inevitable, and in our judgment undesir- 
ij:)le, consequence is the prevalence of " special pre- 
paration " or “ cramming " for this examination. 

We have taken pains to ascertain the extent to 
which cramming prevails, and we find that more 
than half the candidates undergo some kind of special 
preparation for this examination. To some extent 
recourse to the “ crammer is explained by the 
circumstances of boys already in employment during 
the day who have only the evening for study, and 
for whose needs no other agencies suffice. \Ye attach 
greater significance, therefore, to the statistics show- 
ing, for candidates who give their whole time to 
instruction, the percentage who attend places of 

special preparation." 

Such statistics show that the number of candidates 
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who leave school for this purpose is greate. between 
15 and 16 than between 14 and 15 of age; 

greater still between 16 and 17, and greatest betw een . 
17 and 20. 

This tendency is dcublv objecTionable. In the 
first place such special preparation necessarily ^psts 
money, while those boys from the elementary schools 
who are likely to be snccessf li in this examination 
would, in most cases, be receiving j^raluitoiis instruc- 
tion in the public secondary schools. ^Moreover, in 
so far as this system makes success dependent upon 
an expensive substitute for the ordinary school educa- 
tion, it necessarily places the poorer candi^’^ates at a 
disadvantage. 

In the second plac^, the interruption of a^^chool 
curriculum is injurious to the boy’s best interests. 

More general recognition is now being won by the 
great principle that the most valuable elements in 
education those formative influerces which help 
to mould and develop character. Such influences*^'' 
form no part of the atmosphere of a “ cramming 
institution, the main object of which is to surmount 
or evade the difficulties of an examination paper with 
the smallest expenditure of time and trouble. 

We do not deny that useful instruction may be 
given in such institutions. In so far as they supple- 
ment the strictly educational curricula of the schools 
of the country by teaching such technical subjects 
as book-keeping, shorthand, *and typewriting, they 
perform a useful service to the public ; but their 
professed aim is different from the aim of true educa- 
tion. The principal aim to be kept in view in re- 
cruiting the Civil Service (towards which examina- 

83 



EXAMINATIONS— NOTE A 

tions are a necessary, if a somewhat fallible means) 
is to obtain the best results of the regular educational 
system which \s in actual operation ; and, when that 
system becomes sufficiently uniform throughout the 
United Kingdom to permit such a course, we think 
thaj^ Government should require a school-leaving 
certificate as a condition of candidature for the clerical 
examinations. 

(4) First two lines of the fourteenth paragraph of Section 

43 (p. 40) 

43. It is to be remembered that competitive ex- 

amination is a means, though necessarily an imper- 
fect one, of testing natural ability, as well as acquired 
training. * 

(5) Section 44 (pp. 40, 41) 

44. In regard to the suitability of the present 
i^Class I examination, academic opinion was not 

unanimous. There was, indeed, something approach- 
ing agreement in the view that the competitive 
examination is the most, if not the only, practicable 
means of selecting candidates for the situations which 
we are now considering. But the imperfections of a 
written examination as a test, not merely of know- 
ledge, but also of capacity — of the faculty of sound 
judgment, common sense, resourcefulness, and reso- 
lution — were often insisted upon. 

To make good these defects several suggestions 
have been put before us, to two of which in particular 
we invite attention, viz. : (a) the selection by the 
authorities of the various Universities of Honours 
men, also distinguished for their judgment, good sense, 
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and general capacity in stixdent lift:, for ccmpiltion 
inter se for the situations vacant ; and (b) the insiitu.- 
tion of a viva-voce exaininatkm supplementary to the 
written test, witli the object of ascercairnng (.an- 
didate’s general mental calibre . and the possession 
of those qualities of common sense, sound judgment, 
resourcefTilness and lesolution, upon which the 
written paper rarely gives assurance. 

The Vice-Chancellors of the Uni /ersiiios of London 
and Wales entertain<^d a favoii’*able cpinion of the 
former of these suggestions ; indeed they seemed to 
regard it as likely to become the normal method 
of recruitment ot the future. We have considered 
whether it could be appliea as an experimental 
measure to a proportfbn of vaca.xms annually, but 
we are satisfied that the practical difficulties are too 
serious to permit even a partial or tentative trial of 
the system. We are unanimously of opinion that, 
whatcv(^.r the future may have in store for us in this 
connection, the method caniiot generally be applied 
to-day, if only in view of the impossibility of closing 
the competition to non-university candidates. 

(6) First paragraph, and first eight lines of the second 
paragraph of Section 45 [p. 41) 

45. In respect of the second suggestion referred to 
above, we call attention to the opinion which the 
Civil Service Commission entertained on this matter 
in the earlier years of their activity. In their Tenth 
Report (1865) they say of the examination for the 
Indian Civil Service : — In each subject there is a 
viva-voce examination, to which we attach great im- 
portance, introduced for the express purpose of fully 
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testing the genuine character of a candidate's know- 
ledge, in addition to calling into play those qualities 
which are little, if at all, elicited in an examination 
by written papers, viz. his readiness, self-reliance, 
and moral courage.'’ Wc have ascertained from the 
Service Commission that this practice of viva- 
voce examination was continued until the year 1891, 
when it was abandoned, apparently owing to the 
increase in the number of candidates caused by the 
raising of th^ age limits. 

We do not now propose for the Home Service the 
establishment of the viva-voce examination conducted 
by each examiner in each subject taken by a candi- 
date, although some of us believe that such an exami- 
nation would be an effective help towards determining 
the candidate's knowledge of the subject. Our 
object is different : it is to ascertain whether a general 
viva-voce examination based on the whole syllabus, 
and conducted by a single committee of three care- 
fully chosen examiners, might not be made an effec- 
tive help towards determining those of the candi- 
date’s qualities which are little, if at all, tested in an 
examination by written pape^rs. 

(7) Section 46 (p. 41) 

46, Certain academic witnesses seemed to think 
that the present syllabus gives undue weight to the 
studies most commonly followed in the oldfr Univer- 
sities and that the modern institutions are handi- 
capped by it in the competition. Undoubtedly the 
most salient fact emerging from an analysis of the 
academic antecedents of the successful candidates 
is the great numerical preponderance of those who 
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have been educated at Oxford and Ca?ubndge, Ci^pcci* 
ally at Oxford. 

(c-^) Section 50 (pp. 42. 43) 

50. However roncJusive in the opinion ot some, 
md probabU' in llie opinion of others, may bo '*:hcse 
explanations of the facts mentioned in paragraph 46 
above, they do not in our inngment offer a sufFicient 
reply to the inquiry whether ihe conditions of the 
examination, the grouping of the subjects, and the 
relative values assigned to each group, do or do not 
give to any subject or course of study an undue 

advantage in the ex'^mination hall It is, in our 

opinion, highly undesirable that such an inquiry 
should remain unanswered. A sati .factory answer to it 
can be given in only one way, namely, by a full inves- 
tigation into the facts. We were glad to learn from 
the Civil Service Commissioners themselves that they 
concur in the necessity for such an inquiry ; and we 
accordingly recommend the Government to appoint 
a Committee with the object of ascertaining whether 
there is any foundation for or reality in the misgiving 
which certainly y^r^^vails in some quarters that the 
scheme of examination for the Class I. unduly favours 
the curricula of the older Universities and handicaps 
those of the newer. On the Committee the Civil 
Service Commission and both University and School 
experience should be represented, and it should be 
presided over by some man of recognised judgment 
and eminence in public life. Should it be found that 
any change is desirable, then it would be necessary 
for the Committee, while maintaining the high stan- 
dard necessary for the examination, to revise and 
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rearrange the syllabus, weighing, with even balance, 
the educational values of the studies of the older 
Universities against those of the modern learning, as 
it has been called, to which the younger Universities 
are thought to pay more attention. The revised 
syllabus thus made out should be submitted to the 
Gov^ment with whom the decision must rest. 

(9) Sections 4 and 5 of the Summary of Recommenda-^ 
Hons (p. loi) 

The General^ Civil Service, Administrative and 
Clerical. 

(a) General 

4. The competitive examinations for recruiting each 
class of officer, administrative and clerical, should be 
adjusted in respect of the age of competitors and the 
subjects of competition to the stages of the educational 
system actually existing in the country. (Chapter 
lil, § 6 , p. 29.) 

5. The examinations should be directed to testing 
the natural ability of candidates, and the results of 
their education both with respect to acquirement of 
knowledge and the formation of mind and character.^ 
It should not be directed to testing proficiency in 
particular subjects which lie outside the normal scope 
of education, 

1 The Commission give no hint to the Civil Service Com- 
mission as to how examinations are to be directed to, testing 
the formation of character. — P. J, H. 



NOTE B 

ON THE REPORT OF THE TREASURY COM- 
MITTEE ON CIVIL SERVICE, CLASS I. 
EXAMLNATION (1917) > 

The Coininittee reierred to in the heading was con- 
stituted as follows : Mr. Stanley Leathes, C.B. 
(Chairman) ; Sir Alfred Ewing, K.C.B., F.R.S. ; Sir 
Henr}^ Miers, D.Sc., «F.R.S. ; M^. H. A. L. Fisher, 
F.B.A. (replaced on his appointment as President 
of the Board of Education by Dr. W. H. Hadow, 
M.A.) ; and Prof. W. G. S. Adams, M.A., with 
Mr. D. B. Mair, M.A., Director of Examinations to 
the Civil Service Commissioners, as Secretary. The 
reference to the Committee was : 

To consider and report upon the existing scheme of 
examination for Class I. of tne Home Civil Service : 

To submit for the cTmsideration of the Lords Com- 
missioners of His Majesty's Treasury a revised scheme 
such as they may judge to be best adapted for the 
selection of the type of officer required for that class 
of the Civil Service, and at the same time most advan- 
tageous to the higher education of this country : 

And in framing such a scheme, to take into account, 
so far as possible, the various other purposes which 
the scheme in question has hitherto served, and to 

^ [Cd. 8657.] Price ^d. Published by H.M. Stationery 
Office, and to be purchased of any bookseller. 
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consult the India Office, the Foreign Office, and the 
Colonial Office as to their requirements, in so far as 
^ they differ front those of the Home Civil Service. 

The scheme of the Committee may be summarised 
briefly as follows : 

Ihe examination is to be divided into three 
sections : Section A, an examination mainly general 
in character, obligatory for all candidates, with a 
maximum of 800 marks ; Section B, a specialised ex- 
amination, including fifty-nine separate subjectswith 
maxima varying from 100 to 400, candidates being 
‘ allowed to take up subjects in the section up to a 
total of 1,000 marks ; Extra Name rum Section, cover- 
ing 100 additional marks for translation papers in a 
language not already taken by ihe candidate. 

The main xx)ints in which the scheme differs from 
the existing scheme (September 1917) is in the char- 
acter of Section A, and the readjustment of the scale 
of marks in Section B, which is intended to place 
on an equality “ the chief studies which are pur- 
sued by students at the University up to an Honours 
Course, viz. Classical languages, history and litera- 
ture ; Modern languages, with history and literature ; 
History ; Mathematics ; and the Natural Sciences.^ 

Section A includes the following items : (i) Essay 

(100 marks) ; (2) English (100 marks) ; (3) Questions 
on contemporary subjects, social, economic, and 
political (100 marks) ; (4) Questions on general 

principles, methods, and applications of Science (100 
marks) ; (5) Translation from one of a number of 

* To obtain the maximum in Natural Sciences, candidates 
must take one main subject up to the “‘higher level “ and 
two subsidiary subjects up to the lower level, 
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European languages, or Latin (xoc» m<xrk«) ; (6) A 
viva-voce examination (300 marks) To^ai, 800 marks, 
"This Section," says the Repo^, "is intrnded to 
test t ue candidates* know] edge of 1 he English language 
and their capacity for it;, skilfuj use. their accurate 
command of Ircowlcdge v/hicii they should, ..have 
acquired m the course of a systematic education and 
self-edacation and should liave retained to assist 
them in their future work, a, id tiieiT equipment in 
one foreign language at least for v/orking purposes. 
The languages selected are those most likely to afford 
information useful to public servants. As circuifi- 
stances change others shouM be added at the dis- 
cretion of the Civil Service Commissioners. 

" The viva voce i^hould be a by means of ques- 
tions and conversation on matters of general interest, 
of the candidate's alertness, intelligence, and intel- 
lectual outlook, his personal qualities of mind, and 
mental equipment. 1 

"It is not intended that any candidate should be 
disqualified for failure in any of the parts of this 
Section, or in the Section as a whole, but that the 
Section should count substantially in the competition," 
The Section is devised to ensure, as far as possible, 
that gm^raZknowledgcof both science and the humani- 
ties which Huxley, a pioneer in this matter, pointed 
out as essential for every " well-educated man.** 
Before Huxley's time, education meant the humani- 
ties only ; since his time some would have it mean 
" science " only, in a narrowly restricted sense. If 

1 [C/., on this point, section 45 of the Majority Report of 
the Royal Commission on the Civil Service ;i 9 i 4 » quoted on 
p. 85 above.] 
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the Committee's scheme is adopted, the Class I. 
examinations will at any rate be free from the reproach 
that they tend to perpetuate the neglect of science " 
which has been so serious a reproach to our educa- 
tion, and which has had such grave consequences 
in thfi War. The Committee have given specimen 
examination papers ; but the principles to be 
adopted in marking are only vaguely indicated by 
the following rubric : '' Each candidate's answers 

will be valued as a whole, and any candidate who 
presents answers to less than three or more than four 
oi the questions will be presumed to do so in knowledge 
of the value of the answers which he is able to give 
to the questions he chooses." 

The specimen English paper, which is on lines 
largely in accordance with suggestions put forward 
at various times by the present writer, constitutes 
a new departure in examinations of this character. 
It is a test of general intelligence of which the result 
cannot possibly be falsified by special preparation. 

The re-introduction of the viva-voce test, to which 
300 marks are attached, would again, as I have sug- 
gested (pp. 21-22), constitute a real reform. But it is 
one which needs very careful safeguarding ; and I 
am not sure that it would not be well to make it 
public, like so many foreign viva-vocc examinations. 
The presence of strangers who could not influence 
the verdict would add little or nothing to the severity 
of the test, for which cleaiVly the candidates would 
prepare themselves, by undergoing similar tests in 
class before their comrades.^ 

* C/. '* The Aim of the New Curricula of French Secondary 
Schools for Boys," by the present writer, in Special Reports 
of the Board of Education, vol. 24. 
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NOTE C 

THE CONNOTATION OF THE WORD CUL- 
TURE’' (sec als^ Note D) 

There are many interesting passag<?s bearing on this 
point in J. H. Newman’s The Idea of a University, 
third edition (1873), especially in the " Discourse 
on Knowledge viewed in relation to Learning." ^ 
The following (p. is typical of Newman's views : 

" When this analytical, distributive, harmon- 
izing process is away, the mind experiences no 
enlargement and is not reckoned as enlightened 
or comprehensive, whatever it may add to its 
knowledge." 

Newman seems to have in his mind onl^y that colder 
aspect of additions to knowledge by assimilation of 
fresh facts to pre-existing intellectual possessions 
which the modern psychologists call " apperception." 

The following passage from Goethe gives a fuller 
description of the same process, in which the emo- 
tional aspects are taken into account : 

1 The book was originally published in 1852 under the title. 
Discourses on the Scope and Nature of University Education, 
etc., and the particular essay referred f:o was entitled, " On 
Philosophical Knowledge viewed in relation to Mental 
Acquirements.*' 
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Everything of outstanding excellence {Alles 
Vorireffliche) gives us for the moment a sense of 
limitation, b&ause we feel we have not reached ' 
its level ; only in so far as we afterwards make 
it part of our own culture (CuUur) and an element 
*n our own mind and spirit does it become to us 
dear and valued/’ ^ 

There is an interesting parallel passage in E. F. 

Benson’s clever contemporary novel, The Climber : 

« 

* "... but that assimilation which is necessary 

before facts can become the food of culture, 
that kindling of the blood, as with romance, had 
not occurred/' 

I do not quote from Matthew Arnold’s Culture 
and Anarchy, partly because it is so well known, 
partly because I propose to deal with some aspects 
of it elsewhere. 

In the text I have compared the individuality of 
culture with the individuality of conscience, a com- 
parison suggested to me, I think, by an interesting 
book by the French physicist, Henri Bouasse (whose 
constructive suggestions seem to me more valuable 
than his criticisms of his fellow- workers), but the 
comparison ought not to be pushed too far. Culture 
is clearly to be distinguished not only from the moral 
but from the " intellectual ” conscience to which I 
have referred in the Writing of English, p. 63. 

‘ Maximen tind ^flexionen, Goethe Gesellschaft edition, 
1907. P- 65- 
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KVLTUR AND CULTURE 

The history of the Vvord Kidtur ^ in G^Tmany is from 
the point of view of pure semantics almost a para- 
dox ; from the point of view of the history of man- 
kind it is grave and tragical — corruptio opUmipessima, 

Thanks to successive editions of the Dictionary of 
j. C, Adelung, we can safely say that the word 
became of general use in Germany between 1774 and 
1793. It was introduced from the French, and took 
with it the tw^o active significations, the literal one 
of cultivating the land, and^the figurative one of culti- 
vating the mind, without any marked departure from 
the French original,* But in the figurative sense in 
which the word is most generally employed, to signify 
tb© result of such cultivation as applied to man, there 
is a wide and important departure. Whereas " cul- 
ture/' both in French and in English, is always 
used with special emphasis on the intellectual and 

1 Cultur was originally spelt with a C, and is so spelt 
in most of the eaniei* texts. X. H. Campe, however, in 
the posthumous edition of Adelung's Dictionary points 
out that the letter C is unnecessary in German, as it may 
always be re|>laced by K (as in Kultur), by Z ; and this 
method of spelling is now in general use m Germany. 

• Cf. the definition of cuUnre in the edition of the Dictionary 
of the French Academy of 1762. 
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aesthetic aspect of its general connotation (Murray’s 
Dictionary gives as one of the meanings " the intel- 
^lectual side of «ivilisation ”), in German the moral 
aspect is inseparable from the use of the word, 
except in the specialist writings of one or two phil- 
osophers.^ Adclung, who, by his use of the word in 
the compound Kulturgeschichte, and, by his popular 
work on that subject, largely contributed to its accept- 
ance as an equivalent for tlie English civilisation,” 
defines the word as meaning ” the ennobling or re- 
fining of the ^^hole spiritual and bodily powers of a 
’ person or a people, so that the word connotes not 
only the enlightenment (Aufkldrung) and ennobling 
of the understanding by delivering it from prejudices, 
but also the polishing (Poliiur) ^lind refinement of the 
moral qualities (der Sitien)” At the beginning of the 
nineteenth century the word was used, in Germany, 
and it still is used, in the following figurative senses : 

(i) To signify ” state of civilisation ” in its widest 
sense, without any implication of such a state being 
good or bad. It is this sense that it carries in the 
widely used expression Kidturgeschichtc. The 1897 
edition of Meyer’s Konversaiionslexikon states that 
Kulturgeschichte requires an investigation of the 
whole social life, including dwellings, hygiene, clothing, 
furniture and utensils, way of living, food, morals 
and customs, legal conceptions, beliefs and super- 
stitions, at different periods.” The distinguished 
English ethnologist, E*. B. Tylor, in his .book on 
Primitive Culture, uses the word exactly with such 
connotation a^a synonym for ” civilisation in its 

^ For example, H. Rickert, in his Kulturwissenschaft und 
Naturwissenschaft (1910), see pp. 26, 27 and passim. 
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widest sense/’ ^ It seems cleai that Tylor was 
influenced by the German use of word 

‘'•Kultur/' and so gave to the * English wdrd i 
“ culture *’ a sense which it did not previously 
possess, and still only possesses in English as a 
term of cthnoiogv. * 

(2) To mean" civilisation as ^'pposed to barbarism,” 
the " K alt nr ” of people who are r^^garded as 
being on a high plane, GootLe, in a remarkable 
passage in the Conversations with Eckermann (third 
part, March 14th, 1830), contrasts Kultur und Bar- 
bar ei} The sense of civilisation as opposed to bar 
barism is also well illustrated by the following passage 
from Schiller. Speaking of the influence of Italy on 
France in the Renaissance of the sixteenth century, 
he says: "The dawn of CAdtur appeared. France 
was already hastening to welcome her civilisation ” 
(i.e,, after the Dark Ages).* 

From this limited sense of civilisation were de- 

• 

1 See Preface to Primitive Culture, 4th edition. The first 
edition appeared in 1871. 

2 But a little later in the same passage Goethe uses it in the 
first and more general sense, and says : “ There is something 
singular about national tfatred ; one always finds it strongest 
and most violent at the lowest stages of civilisation {Kultur).** 
Goethe is explaining to Eckermann why he declined to write 
songs of hate against France, and asks how he could write 
songs of hate without hating, and how could he, to whom 
culture and barbarism were the only things that mattered, 
hate the French, who were ampng the most cultivated of 
people, and to whom his own culture (here he uses the word 
Bildung, see footnote on p. 99 below) was so largely due. 

® " Die Morgenrdthe dev CuUur erscl^ien. Schon eilte 
Frankreick mit schnellen Schritten seiner Civilisirung ent- 
gegen** {GeschichtS dev franzdsiscken Unruhen, etc., K, 
Goedeke's edition of Schiller’s Works, 1857, vol. ix. p. 294). 
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veloped three more specialised uses of the word, 
set out below. 

« (3) Moral elevation. Goethe writes in his Maxi- 

men und Reflexioncn (Anhang, Cotta edition, vol. iv. 
p. 230) : Piety {Frommigkeit) is not an end in itself, 
but a means by which to attain through the purest 
spiritual quiet to the highest moral elevation (Culiur)/* 
We find the same use of the word in Dichtung und 
Wahrheii, Part II. Book ix. (Cotta edition, vol. xxiii. 
p. 188). Goethe also sometimes uses the phrase 
sittliche Kulfur” (see, e.g., Dichtung und Wahtheit, 
Part II., Book vii., loc. cii., p. 9). 

(4) Individual refinement and sensitiveness, com- 
bined with knowledge, in the intellectual and aesthetic 
spheres. This is approximately the sense in which I 
have used culture in the text. It is a rare sense 
in German. An example of its use by Goethe is 
given in Note C, p. 94, above. 

(5) External polish, relating to dress, manners, etc. 
It is in this sense that Goethe uses it in Faust (Part I. 
lines 2495-6), where Mephistopheles explains to the 
Witch his reason for concealing his horns and tail as 
out of fashion. 

“ Audi die Cultur, die alle Welt bedeckt. 

Hat auf den Teufel sich erstreckt." 

which may be freely rendered : 

“ Of culture too, which all good folks now wear. 

The Devil himself lias got a share.*' » 

With a word of such wide connotations individual 
differences of usage by different authors would in any 
language be inevitable, but the central fact in the 
history of the wprd “ Kultur ” in Germany is that 
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the most frequeot and commoh sense ii* wMch the 
word is used is that of civilisation as opposed to 
barbarism, and with stress laid on 'he moral jide 
civilisation.' Absurd and ndiculous as it man seem, 
it is quite clear that it is hecanse of this sense that 
the Pan-Gtiman^ in iheir own country have been 
able to use it as signal and a watchword almost 
comparable to the Liberty, E.pialit}/, and Fraternity 
of the wars of the French R(‘VoIation. To us it nov/ 
stands for the massacre and enslavement of Belgium, 
Serbia, Armenia, the repudiation of aJ international 

» It has been stated that the proper trr.nslation of “ cul- 
ture " in tliC L'nglish sense is BiUhnig, and not Kultur. Dii- 
dung is a synonym of the fourth sense of KuUur, which is 
rare in German, and to which it corresponds almost exactly. 
Thus F. Paulsen says of Bildung that it consists not merely 
in the possession of things known (Keuntnisse) but in the 
possession of the living forces of active knowledge {des Er- 
kennens) and of action in which the inner living individual 
(die inner e Lebensform) takes p|Lrt (Article on " Das Moderne 
Bildungswesen " in P. Hinneberg's Die Kulti.r der Gegen- 
wart (1906). Goethe, however, seems to limit the word 
Bildung rather to a receptive assimilation of knowledge than 
an active one. 

It has also been stalet^ that Zivuisation in German means 
only material civilisation. That this is not strictly accurate 
is shown by the quotation from Schiller (on p. 97 above). 
Hamack, the theologian, in a passage written in 1909, says 
that England’s “ Zivilisation " is, taken as a whole, still 
superior to that of Germany, when, from the argument, it 
is quite clear that he means something much wider than 
material civilisation (A us Wissenschajt und Leben, 1911, 
p. 200). On the other hand, W, Muench regards Kultur as 
consisting of “ internal Kultur/' often called Kultur, and 
“external Kultur,'* called " Zivilisation," the two being the 
inseparable wings of a “ single front “ (Aus Welt und Schule, 

1904)- 


99 



EXAMINATIONS— n6teJD . 

law, tfte atrocities of Louvain and Rheims. The 
moral degradation of Germany has degraded the word 
* itself. The degradation has not taken place without'a 
long struggle, and the uses and definitions of the word 
Kultur by men of different political schools, Liberal 
and Socialist on the one side, and Pan-Germanists 
on the other, are historically significant of that 
struggle. Harnack contrasts ironically the Kultur 
of the present day, with its arriere-pensSes^ and 
compromises and concealment of its original source, 
with the spirit of scientific investigation {Wissen- 
schaft) which knows no compromises and no arrihre- 
pensees, and suggests that, if things had to be settled 
by contemporary diiithoniy, Kultur would be entrusted 
to “highly trained journalists*” while Wissenschaft 
would be shut up in a monastery.® 

Paul Michaelis, a liberal journalist advocating a 
closer union between Germany and England in his 
book on Prussian Culture,* actually introduces into 
his definition of Kultur' ihe phrase “ the will for 
Peace.” But Bernhardi, in his book on the Future 
of Germany, published in 1913,* demonstrates that 
an essential part of KiilHir is the will for War, his 


1 Harnack uses the word Riichsickt ; I know of no English 
equivalent for the word in this particular sense. 

s Reden und Aiifsdtze, 1904. ii., p. 371 ; the passage itself 
dates from 1905. Harnack, alas, signed the Chauvinist mani- 
festo of the German professors in 1914 to which the noble 
and moderate reply of the French Universities forms so 
striking a contrast. 

3 Von Bismarck bis Bethmann. Die PolUik und KuUur 
Grosspreussens, 1911, p. 271. 

* Unsere Zuhunft, see pp. 40-43. The book has not, so 
far as 1 am aware, been translated into EngUsh. 
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f 

demonstration beuig based on the theory thaf Ger- 
man KuUur h the most necessary r^ctcr in human 
progress/' and that the ricquisitjoif of colonies by^ 
Germany is essential for the propagation of Cermau 
Kultuf, '‘KuJtur'* is th* R<d C’‘os? fag under 
which the Gerirrin pirates have sailed to the attar.k 
on hamanity. 

The obiert of this nole Las been nartly to clear 
up some current misconceptions, oat mainly to show 
that our fine word ** culture/' which lias no synonym 
in English, ought not to be driven from our language 
by the degradatktn of Kulinr. 1 hope to deal with 
the question more fully elsewhere. 
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NOTE E 

PROF. EDGEWORTH'S INVESTIGATIONS ON 
THE STATISTICS OF EXAMINATIONS 

1. The only ^ablislied statistical investigations of 
importance on the results of examinations of which 
I am aware, are contained in three memoirs by Prof. 
F. Y. Edgeworth, of the University of Oxford ; 

(i) The Statistics of Examinations, Journal of the 
Royal Statistical Society, vol. li., pp. 599-635 (1888), 

(ii) The Element of Chance in Competitive Examina- 
tions, id,, vol. liii,, pp. 460-75 and 644-63 (1890), and 

(iii) On Problems in Probabilities [Philosophical 
Magazine for August, 189?)).^ 

2 . Definition of Standard or True Mark and general 
Method of Investigation . — In comparing the marks of 
different examiners with one another, it is necessary to 

‘ These papers also contain references to other works by 
the same distinguished author, including a treatise on the 
art of measurement, entitled Metreiike (London : Temple Co,). 
An elaborate account of the mathematical theory of Proba- 
bility is given in Professor Edgeworth’s article on that sub- 
ject in the eleventh edition of the Encyclopadia Britannica. 
The first two papers, which are summarised below-, are intended 
for the general reader ; the third contains a more technical 
treatment of problems dealt with in the first two. The 
present writer has no expert knowledge of statistics, and is 
greatly indebted to Professor Edgeworth ^o^ having revised 
this summary. 
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assume that there is i^ome standard, or ** true / mark. 
In the second paper, Prof. E. postulates *' that the 
true or standard mark of any piece ol work is the 
average of the marks given, by a Jarge number ot* 
competent examiners equally proficient in the subject 
and instructed as to the character and purpose of 
Mie examination, ’ ^ 

Prof. E. clearly states this as a postulate, not as 
an axiom ; but it is a convenient postulate, and is 
probably geneiallv acceptable- fhe general method 
adopted by Prof, E. is to ascertain what this standard 
is in a number of typical cases, and to investigate ' 
the divergency from this standard, or error, of the 
marks allotted by individual examiners, and to deduce 
general laws of such, errors, with the object of ascer- 
taining the probability that u,n error of assigned 
extent will be committed by a competent examiner 
in marking a single piece of work of this kind ; and 
from this to deduce the probable errors in the more 
complex cases which present themselves in practice.* 

1 This is a development of the corresponding statement in 
the first paper. 

3 Even at the outset there are at least two practical diffi- 
culties to be got over : J^a) the difficulty, involving as a rule 
expense, of obtaining a large number of “ equally competent 
examiners to look over and mark the same piece of work ; 
and (6) the difficulty of obtaining any definition of the " pur- 
pose " of the examination in question. As I have pointed 
out in the text, this absence of definition of purpose con- 
stitutes, in my judgment, one of the great weaknesses of 
our present system, and one wMch necessarily leads to far 
greater divergencies of marking than would otherwise exist. 
Prof. E., in his second paper, expressly states that his subject 
is '* not the philosophy of examination, but the statistics of 
marks,*' and docs not attempt therefore to go behind the 
marks allotted. 
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* # 

3« Ii^iual compensation of independent errors , — 
Effect on aggregates of marks. — Prof, E. points out 
that, according Jto the Calculus of Probabilities, if a 
*certain quantity is liable to a probable error of i per 
cent, in its measurement, the probable error on the 
aggregate of the sum of a hundred such quantities 
is*not I percent., but only cent., that is i per 

cent, divided by the square root of loo, in virtue 
of the mutual compensation of independent errors.'' 
The statistical theorem of which this is an example 
is much used throughout his investigations.^ (See 
• also § 24 , p. 130.) 

4 . The above considerations are common to the 
two papers, of which I now submit separate sum- 
maries, necessarily brief and imperfect, but bringing 
out certain salient facts. 

^ The whole investigation is based on the Mathematical 
Theory of ProbabiUties. The following definitions and state* 
ments will be useful to the reader : — 

(1) The probable error of a series of observations is that 
divergence from their mean on either side . . . within which 
exactly half the observations lie. An approximation to the 
probable error for a given series of observations is obtained 
by arranging all the observations ki order of magnitude; 
marking the magnitude, say, a, above which 25 per cent, 
of the observations lie, and the magnitude, say / 3 , below 
which 25 per cent. lie. Half the difference between a and 
/9 is the probable error. 

A useful way of illustrating this is to say that if one 
observation is chosen at random out of a group it is as 
likely as not that it will not lie further from the* average 
than the probable error. (A. C. Bowley's Elements of 
Statistics, second edition, 1902, pp. 281-2.) 

(2) If n marks (or other magnitudes) are arranged in 

an ascending order of magnitude the mean of the - mark 
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Paper I ’ 

6. Choice of an Ideal Typical Examimiion fdr 
Investigation . — ^In order to make tfie subject tnoif 
readily inteiligible. Prof. E. fakes as a particular case 


and the (— I 1)“* mark is called the lower quartile ; the mSan 
of the ana ^he (--4-1)**' marks is called the mean of 

the set ; and the mean of the f— V‘' marks 

V 4 / V 4 

is called the upper quarti’e. The probable error is thus half 
the difference between the upper and Icwer quartiles. 

It will be seen -Jhat the '* possible error is always con - 
siderably in excess of the probable error. 

(3) The " mean error of a series of n observations of 
which the individual errors or differences from the mean 
value are ei, cz, . . . e«. is the square root of the sum of 
the squares of those errors divided by their number, i.e., 


V' 


ri* 4- <?2* 4- ^3* 


4 - 


For aggregates fluctuating according to the ordinary " law of 
error " the " mean error," so defined, is about i’25 times the 

average error, i.e. lljl— ^ each e being taken 

positively, or in absolute quantity, as the phrase is, without 
respect to sign. • 

The " frequency curve " of a series of magnitudes (such 
as a series of marks or errors in marking) is a curve of which 
the abscissae represent the magnitudes and the ordinates 
the comparative frequencies with which the corresponding 
magnitudes occur in the series. The frequency curves for 
errors in marking (and especially for errors on aggregates of 
marks, see § 17 below) appear in general to belong to the 
special type called the curve of error, of which Prof. Edge- 
worth Ukens the shape to the longitudinal section of a 
gendarm&*s hat. 

(2) and (3) areTderived from Prof. E.'s papers. 

^ For Summary of Paper II see p. 118 below. 
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a typioll examination in which candidates are 
arranged according to the aggregate of marks which 
they have received on ten different papers, for each 
of which 300 marks are allotted, and assumes “ that 
those who, without failing in any subject, have not 
obtained in all the subjects more than 2,000 marks, 
are placed in alphabetical order in the category of 
' Pass ' ; and those who have obtained more than 
2,000 are placed in the category of Honours and 
arranged in ‘ order of merit * according to the number 
^ of marks received/’ The paper investigates the 
'error to which these aggregates of marks are liable ; 
and thence proceeds to deduce the degree of certainty 
incident to the distinctions grounded on these marks. 

No attempt is made to estimate the chances intro- 
duced by the possibility of a candidate being out 
of sorts ” on the day of examination, or of the paper 
being specially adapted to the accidents of his taste 
or reading ; it is assumed, as in the case of our 
public examinations may be safely assumed, that the 
paper set in each case is an ideally fair one."' 

6 . Errors due to Minimum sensibile. — Prof, E. next 
points out that there must be a minimum sensibile 
in our perception of excellence, *^of which the magni- 
tude will vary with the subject, being least in Mathe- 
matics and most in subjects where marking depends 
greatly on “ taste.’' 

Where subjects are for the most part of the simpler 
sort, Prof. E. estimates t^he “ probable error of the 
mark for each answer at 5 per cent. ; in other words, 
when an examiner is putting down 20 (the maximum 
being larger) he very generally hesitates between 
19 and 21 as equally representing to‘him the worth 
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of the ansv;er. Prof. E. then supposes iiiat the 
average number of questions attempted So. and 
deduces that the probable error 'Zander th' > head, 

is at least percent, or } per cent, (see § S^bove),' 

Thus if marks between 2,000 arid 2,050 had been 
assigned tc a candidate, there is an uncertainty on 
thL account represented by a probable error '' of 
10. If the aggregate of marks ^olbns what is oiten 
called the normal law' (cr irequency curve), this 
corresponds to a possible error*’ of 20, the odds 
against such an error being only 6 to i, and a ” very 
improbable error " of 40, against which the odds are 
250 to T. Even or the most limited view of the 
influence of chance a considerable uncertainty 
arises as to the true position 01 candidates in the 
Honours list. It is to be remembered that this 
error is altogether unavoidable. 

7 . Errors due to Personal Equation of Examiners. 
Variations of two kinds . — much more serious cause 
of inaccuracy is the ” personal equation ” of the 
examiners. Two species of variation are to be dis- 
tinguished under this head, (i) variations relating to 
particular answers, and (2) variations relating to the 
whole scale of marking. 

i Thus in a pass examination in which pass or failure de- 
pends on the result of two papers, and say sixteen answers 

in all, the probable error wovld be — or 1*25 per cent. 

1 cannot help regretting that the author did not deal more 
completely with the simple case in which " pass ** and 
" failure/' depend on the results obtained in separate sub- 
jects, independently of the aggregate of marks obtained in 
all subjects. 
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Prof, E, states that his data (which are not quoted 
in full) sljow that the observed mean difference per 
cent, between two (numerous) sets of marks is 20, 
Taking into account an error of 5 per cent, due to the 
existence of a minimum sensibile, he finds the probable 
error due to the first cause to be 10 per cent. corre- 
sponding to a just possible error of 30 per cent.’'). 
In the typical examination dealt with the probable 

error will be i.e. ri per cent.^ 


^ Prof. E. states that an experienced Oxford examiner 
tells him that, in general, where examiners are careful 
and experienced, marks run very close ; within (say) 
10 in translations (full marks being 100), within a 
class, or, better half a class, in Subjects and essays 
(there being four classes, and the mark given being 
one of the letters a, p, y, 8, qualified in seme cases 
by a + or a — to indicate a doubtful or intermediate 
position ”). 

Prof. E. gives, as an illustration of the second kind 
of variation, the difference of scale, a case of an 
examination in English in which he and a colleague 
“ had agreed beforehand as to the worth of the 
different questions, and as to the degree of proficiency 
to be expected of the candidates," yet the average of 
his colleague’s marks came out 13 per cent, higher 
than his own, the former being 165 (out of a maximum 
of 300) the latter 145.2 Thus, if the examinations had 
been conducted on one occasion by the first ej^aminer 


‘ If, as before, we take only sixteen answers, the probable 
error under this head will be 2*5 per cent. 

• The percentage refers to the mean of the two averages, 
viz. 155. (Footnote by Prof. E.) 
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and on another by tiie second, there would h?-ve been 
a difference of 13 per cent, in the average marks 
allotted. 4 ^ 

Prof. E. suppos es that .in so well organised a 
system as that of our public examinations, care is 
taken to avoid this dicturbanco, ' and that there, is 
a traditjDn as 10 the relative value of each subject 
to be preserved by tne slowly changing staS of chief 
examiners/ ^ 

Still he says, ‘'in spile of ccnscions efforts to main- 
tain con'^ancy, it Vvould not be unreasonable, I think, ^ 
to assume a probable error of 4 or 5 per cent, due to 
the factors under consideration. Thus, supposing 
there was no real difference between tne papers in 
Mechanics of this ye^.rand the preceding, that the real 
average, so to speak, of both sets of papers was 150, 
yet in the sortition of examiners it is likely as not that 
the average may be pushed up to 156 or down to 
144 ; and it is possible that it may deviate much 
further from the ideal mean.'" 

Prof. E. adds in a footnote that, in making this 
estimate, he stands astonished at his own moderation, 

“ for there is no doubt that at many of our public 
examinations the fluctuations in the scale are enor- 
mous.'" He quotes Dr. Venn, ^'an experienced 
examiner, who is also one of the highest living authori- 
ties on statistics/' as saying, “ I have frequently 
raised or depressed my own marks (or my colleagues') 
by as much as 25 per cent, all through in order to 
bring them into general harmony." 

i I presume that Prof. E. refers here to such examinations 
as Class I. Civil Service Examinations, where not only candi- 
dates, but " subjects ” compete. 
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Adopting, however, the probable error of 4*5 per 
cent, in jespect of personal equation, the probable 

%rror on 10 papers will be or about 1*5 per cent.' 

An increase or diminution in the scale of marking 
in a given subject will not affect the relative order of 
candidates in that subject ; but it will obviously 
affect the relative order of the aggregates of marks 
obtained by candidates of whom only some take this 
subject, and therefore their relative order in the 
Honours list .2 * 

Prof. E., by reasoning which he does not give in 
full, comes to the conclusion that the additional 
probable error on the aggregates due to difference of 
scale is only from J to J per ceht. ‘‘ and may safely 
be neglected.” 2 

8 . Errors due to Correction at great speed — or the 
“ negligence of haste.** — He next investigates the sub- 
ject of differences in marking due to difference of 
speed ; and finds that the nfinimumsensibileiox papers 

45 

1 If there are only two papers it will be or about 3 
per cent. 

* Prof. E. does not deal with the question of how far 
the pass in each subject is affected. Suppose the pass mark 
is 33 per cent., the probable error on the 33 marks out of 
100 is 4, As I have said elsewhere, I find it difficult to attach 
any definite meaning to such a pass-mark as 33 per cent, 
on capacity tests, although it has a definite meaning in 
memory tests (see p. 63). It ought to be said that in many 
examinations, probably most, border-line cases' receive 
special consideration which must reduce the probable error 
near the border-line, though it clearly cannot be eliminated, 
since the judgment of different examiners is bound to vary. 

* As will be seen from § 10 below, Prof. E. finally decided 
at a later stage not to neglect this error. 
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corrected at top speed becomes considerable. « Taking 
this into account, he finds a probable error aug to ‘ ' neg- 
ligence of haste " of about 25 per cent. If there a,r^ 
80 items affected by this errar, the probable error Of 


2 < 

the aggregate is about 2'8 per cent. But the 

author assumes that the results in public examina- 
tions are. when Honours are concerned, subject to 
comparath ely leisurely revision and re-examhia- 
tion by certain chief examiners, ' and therefore does 
not take the item into account in dtialing with the ^ 
typical examination under consideration . He regards 
rapid examining as safe enough for the purpose of 
a Pass examination/’ 

9 . Errors due to Fatigue of Examiners. — Prof. E. 
next investigates the error ot marking due to 
the fatigue of examiners, which might be avoided 
if more time could be allowed. He thinks this may 
amount to from i to 2 per cent, on each paper ; 
allowing 1*5 per cent, and 10 papers, the probable 


error is 


-/lO 


or say i per cent, on the aggregate 


marks for the examination. 


10 . Summary of Efrors due to marking in Typical 
Examination. — For the typical examination Prof. E. 
sums up his results as follows : — 

Percentage 
(probable) Error 
of Aggregate. 


Due to existence of a minimum sensibile . . i 

Due to the examiner’s idiosyncrasies : 

(1) as regards particular questions . • li 

(2) as regards scale . . . . 

Due to the “ negligence of examiners " as 

regards scale 


The total probable error expressed as percentage of 
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the aggncgates of marks is the square root of the sum 
of the squares of the separate items, or rather more 
ihan 2 percent.* 

** For instance, if a candidate obtains an aggregate 
of 1,965 marks, thus falling below the Honour 
line by 35 marks, the odds that he ought by 
right to be in the Honours Class are consider- 
able, say one in four. Even if he has obtained only 
1,800 marks, it is by no means certain that he is 
rightly placed among the rank and file of Pass 
. candidates.** 

The same doubt exists as to the rightful position 
of those who are the same distance above the Honour 
line. “To such an extent are the Honours shuffled 
by the fortune of examinations!** 

11 , Comparison of foregoing Results with certain 
Statistics of Cambridge Classical Tripos. — From 
an investigation of the marks of a Cambridge 
Classical Tripos examination, Prof. E, comes to 
the conclusion that th^ probable error on the 
mark of a single paper is not less than 9 per 
cent., and on the aggregate mark of 9 or 10 papers, 
3 per cent. 

Prof. E. gives an interesting table, too long to quote 
here, of the aggregates of marks for ten papers 
assigned to fifteen candidates by two examiners in 
each case. The greatest difference between two 
aggregates is 59, the two aggregates being 472 and 
413 ; the least differenceisS, the twoaggregates being 
632 and 624. Calling the difference between the two 
aggregates in each case and their mean the “ devia- 
tion,** we find the average deviation or “ error “ per 
cent, (of the mean marks in each case) to be 3*5. The 
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‘ probable error ** is *84 of the average error 01 
about 3 per cent. (See also § 19 , p. 125). . 

12 , Error due to possible non-repfesentaiive Char'^m 
acter of ^Answers of Candidates — Total Error of MarUs 
in Typical Examination. — ^Prcf. E. touches on the 
difficult problem of evaluating the eiror committed 
in taking the answers of the candidate as representa- 
tive of his proficiency (in other words, using the 
language of the text, what is prolability of the 
sample which he has given of his proficiency being a 
fair sample). He estimates the variation of marks ^ 
from ansvrer to answer to be adequately represented 
by a probable error of 25 or 30 per cent. ; and on this 
basis calculates the probable error on the aggregate 
of marks for typical* examination to be about 2 per 
cent. 

Cumulating this error with the account already 
furnished, " we find for the aggregate of marks, con- 
sidered as a test of the candidate’s real proficiency, 
a probable error of about ^ per cent., corresponding 
to a possible error of more than 9 percent. It is to 
be remembered that this estimate leaves out of account 
many elements of uijcertainty which are too subtle 
for the Calculus of Probabilities to handle; such as 
the variation of the candidate's spirits, or the circum- 
stance that the questions are not fairly representa- 
tive of the subject. ... It is to be remembered, 
also, that the error in appreciating the answers 
is reck&ned from the sup’posed ideal mean which 
would be presented if a large number of com- 
petent sets of critics were to examine the work. 
If it is asked, what difference is to be expected 
between the results of any two sets of examiners, 
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' ' < 

the estijuate of probable error must be ^somewhat 
increase^/* 

* 13 . In how nuny Cases may award of Honours in 
Typical Examination ha due to Chance? — Prof. E. next 
touches on the question (i) in how many cases 
tjie obtaining or not obtaining Honours may be re- 
garded as accidental, and for this purpose makes the 
assumption, which he thinks legitimate, that out of 
every i,ooo candidates, there are among those who 
have passed, at least 120 who have obtained marks 
between 1,800 i..nd 2,000. On this basis he calculates 
that the average number of candidates who are placed 
in the wrong category owing to chance is about 30, 
although in one year it might be only 20 and in another 
rise to 40. ^ 

14 . Recommendations for Reform : {i) Honours Lists. 
(2) Working of Assistant Examiners. (3) Question of 
making Examination-marks conform to a given Fre- 
quency Curve. — ^To mitigate the defects arising neces- 
sarily from chance, Prof. E. makes certain recommen- 
dations based on purely statistical reasons,'* which 
he is quite prepared to find may be overborne for 
purely practical reasons. 

(i) Arrangements of Honours Lists. — He suggests 
that “ the arrangement of an Honours List in an 
ostensible order of merit should be abandoned when 
the distinctions thus drawn do not correspond with 
any degree of probability to real differences. The 
extent of each homogeneous alphabetically arranged 
class should be considerably greater than the probable 
error incident to the aggregate of marks upon which 
the arrangement is based. Thus, in the example 
chosen, if an aggregate of marks between 2,000 and 
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2,100 is Ijdble to a probable enor of 50, thi^n all or 
most of the candidates between these Uiaits.mifht be 
placed in one homogeneous class.' • » 

(2) QuesUon of Assistant suggestii 

that the functions of Assistant Fxamuiei:; requires 
special consideration, and says there appear to be 
two distinct theories, [a) ancf (ft), with regard tc the 
arretngen ent in respect of H^onoiirs candidates who 
[are reported by the assistant oxenuners to] have 
passed ; and that the two tneories may a omit of 
intermediate shades and practical compromises. 

(a) According to theory {a) the function of assis- 
tants is merely to select provisionally, for careful re- 
examination 7 ft initio by the chief examiners, candi- 
dates who may be deserving of Honours. 

In this case, the papers selected for re-examination 
should be defined, not by the Honour line of 2,000 
marks, but by one considerably below it, due account 
being taken of the ex hypothesi rough marking of the 
assistants being affected by a very large probable 
error. Prof, E. thinks that, according to this view, 
in order to avoid mistakes it would be necessary for 
the chief examiners^to re-examine completely and 
de novo the papers of about half the candidates who 
have passed. 

' Professor Arthur Schuster, on the other hand {Univer- 
sity Review, May 1905), has pointed out that the differentia- 
tion into classes without any indication of order of merit, 
may make an unjust distinction between the last in one class 
and the first in the next. The creation of a large number 
of classes in a typical examination such as that postulated 
by Prof. E., with a class in jilphabetical order for each multiple 
of 100 marks, mi|jht be a satisfactory compromise ; it would 
not be necessary to make use of the word class, the candi- 
dates within each century would merely be bracketed. 
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f 

( J) According to this theory, the marks aj^signed by 
the assistant examiner *'are final, subject to the in- 
spection of the diief examiner, and the revision of ex- 
ceptional cases/' According to this principle, '' or any 
intermediate plan in so far as it involves this principle, 
it*may be expected that the sources of error which have 
been above described under the head of negligence ^ 
will become peculiarly operative/' 

Prof. E, at once mitigates this criticism by a foot- 
note, in which he says : What maybe safely affirmed 
, of any public eltamination is, on the one hand, that 
any method which has commended itself to many 
able and experienced examiners is fairly good ; but 
that, on the other hand, it is not likely to be the 
best possible, if it has been adopfed without reference 
to the theorems in Probabilities which evidently 
dominate the whole subject." 

In order to avoid the error due to marking at high 
speed, Prof. E. next investigates the possibility of 
combining rapid dealing 4 nd accuracy by omitting 
the correction of half the answers in each subject 
and comes to the conclusion that the probable error 
incident to the aggregate of .marks obtained by 
doubling the marks so allotted (the correction being 
leisurely) is only 1*4 times the probable error (and 
not twice the probable error) of the aggregate obtained 

1 I think the word " negligence/' though used by Prof. E., 
implies a moral stricture on the individual examiner or 
assistant concerned which is not intended by the author, who 
points out. in dealing with this matter, that “ the number 
of the staff in relation to the quantity of work to be done, 
and the time for doing it, is sometimes such as to necessi- 
tate an extraordinarily rapid rate of examining/* Prof. E. 
tells me he concurs.. 
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by corre<tting all the answj^rs, the c<>rreetif>n Leing 
leisurely.' 

Thus if '' the total aggregate of :mrks, coiiwuderefi 
as representing tht: real proficiency the candidate 
is liable to an error of 2 or 3 per c( *il. [sec § 10, above] 
the error incident io an examination of half the 
answers is ouly 3 5 or the number of answers being 
large, ard the half selected at random.'' 

Prof. E. suggests, as a simple -nattlud of diminish- 
ing the time taken over correciion which would avoid 
hasty correction is to increase the number of assist- 
ant examiners, aiid further that Ihe answ^ers of the 
candidates should be separable, the answers to (say) 
the first two questions of all the candidates being 
looked over by examiner A, the answers to the second 
two by examiner B, and so on. “ By this arrange- 
ment the influence of the personal equation, intro- 
duced by a plurality of examiners/' would be so dis- 
tributed as to become insensible. ^ 

(3) Question of making Examination-marks conform 

1 This arrangement is adopted in certain examinations. 
It has the further advantage that the exainin£r, who neces- 
sarily marks slowly at fifst, and increases speed as he becomes 
more familiar with the difficulties of the paper and the 
problems involved, attains the maximum speed consistent 
with care more quickly than if he has to deal with, say, four 
or five times as many different questions. The disadvantage 
is that he cannot judge of any candidate's work as a whole. 
It somepmes appears from the correction of a whole paper, 
in a way that correction of a part would not reveal, that 
the candidate is making correct use of terms which he does 
not in the least understand. Hence it would appear, as Prof. 
E, points out, that the principle would require " to be tempered 
with practical s%acity and knowledge of details" before it 
could be immediately applicable. 
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to a give^ Frequency Curve, — Prof. E. suggests that it 
may be possible, in each subject, to determine what 
pught to be the f>roper average for a set of marks of 
different candidates at a given examination and the 
proper*' dispersion " of the marks about this average ; 
and he shows how another set of marks may be 
reduced so as to conform to this arrangement. The 
suggestion is repeated in the second paper in respect 
of Government examinations (see § 26 below). ^ 

* Paper II 

16, The Differences of marking between different 
Examiners, in respect of one Paper, are similar to the 
corresponding Differences in respect of another Paper 
in the same Subject. — Comparison in the case of different 
subjects {Geometry and History). — After certain pre- 
liminary matter (see § 1 above) the author disQUSses 
the general question of laws of error attaching to 
different kinds of judgments or estimates, relating 

1 111 revising this abstract, Prof. E. draws attention to 
the great importance of paying attention to " dispersion " 
when marks arc combined, and points out that if the award 
of a prize is decided by the sum of marks obtained in latin 
and Greek, and if, while the average obtained in the two 
subjects is the same, the “ dispersion " is much greater in 
Greek, the Latin marks will tend to become inoperative. He 
illustrates the argument by curves which cannot be repro- 
duced here; but it is easy to see that if the marks of, say, 
the first half-dozen candidates lie close together in Latin, 
but are widely dispersed in Vireek, the Greek marfes would 
decide the matter. This result would be unfair if it were 
due only to the difference of idiosyncrasy of the Latin ex- 
aminers and the Greek examiners in tlieir method of assigning 
marks. In such a case verbal discussion •of the results is 
essential for a fair award. 
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to things other than examination answers 
weights and heights. He discusses the^ case of 
marks awarded by sixteen difltjfent coiroetent 
examiners to the work in gather: at ico ot a pupS 
in the sixth form of the Noith Londori Cciegiate 
School, and niids the probable crroi deduced 
irom the tixteen observations to be 4*5 per cent. 
" For the eight first in the order of entry it is 3*5 ; 
for the second batch it is 4*5. Accor-''’ ingly, it is net 
a leap in the dark when we Infer that what is true 
of one Set of sixteen examiners marking that paper 
would be approximately true of another set 01 
equally competent examiners.” He finds, furthci, 
that the law of dispersion for two dij^erent papers 
corrected by the same examiners is also approximately 
the same. Although he expected the variation in 
the marking of history papers to be greater than in 
the case of mathematics, the limited experience 
afforded by three sets of marks assigned by nine 
different examiners to three history papers showed no 
material difference between the two cases. 

16 . Marking of piece of Latin Composition by twenty- 
eight differeni Examiners. 

I think it w'ell to quote the following passage 
verbatim: — “The experiment which 1 shall next 
adduce is in alia materia. It relates to a piece of 
Latin prose which the Editor of the Journal of 
Education kindly allowed me to insert in a recent 
number of that periodical, with a preface which it 
may be well here to reproduce in part : — 

‘ I propose, through the medium of the Journal 
of Education, to invite any competent person to 
assign a "mark to the subjoined piece of Latin 
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prose, upon the supposition that he is ^marking 
theVork of a candidate for the Ii/dia Civil 
Service. Let it be distinctly’understood that 
in giving his maijk the examiner is not to 
look to, or wish to illustrate, his own ideal of 
classical elegance, nor yet the degree of pro- 
ficiency which may be current in the school or 
other institution with which he may be con- 
nected, Let him imagine that he has been 
appointed examiner in Latin for the India 
Civil Service, and let him give his mark 
having regard only to what may be expected 
from a candidate for that prize. Let loo be 
the maximum attainable by any candidate. 

‘ To avoid accidental divergence as much as 
possible, to perform the experiment under the 
most favourable conditions, I would suggest that 
the examiners should consist of a pretty homo- 
geneous class — of much the same class as those 
who actually conduct, our public examinS,tions. 
To be more definite, I would invite to take part 
in this experiment only those who have taken 
high honours in classics at one of the univer- 
sities, or classical masters of the sixth form in a 
public school. All such are earnestly invited 
to examine the accompanying piece with as much 
care as if they really were exercising the function 
of a public examiner ; and to send to the editor 
their verdict, guaranteed by their name and 
status, which, it need hardly be added, it is not 
intended to publish. It is desirable that the 
examiners should assign their respective marks 
independently, and without mutugfl conference.' 
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• 

"In jfesponse to this appeal twenty-eight highly 
competent examiners were so kind as to^^niark the 
piece of Latin prose. The groupm| ot these wenty- 
eight indepe -den: marks njay be ^.<:pressed by tile 
accompanying scheme of figureF. 7/ith t]:e iaterpro- 
tation of whici the reader will by this time ^be 
familiar^ . 

II V VII vii V II 

63 6 735 7^'5 ^^5 93'5 

"The actual mark? are : — 4 ^, 5% 67, 67*5, 70, 70, 
72 ‘ 5 > 75 ^ 75 > 75 , 75 . 75 . 75 . 77 , So, So, 8o, 8o. 8o, 8^. 
82, 85, 85, 87*5, 88, 90, 100. 100. 

" If will be seen that 78*5 is the central value, the 
presumably correct mark ^ ; the probable error on 
either side of the correct marK is 5. It may excite 
surprise that this error is not very much greater 
than what we observed in the case of the geometry 
and history, which, without offence, may be described 
as not requiring on the, part of the examiner such 
an exercise of literary taste as the Latin prose. But 
when we consider the extreme as well as the central 
portions of the Latin prose group, we shall be justified 
in attributing to the marking of the more advanced 
work a larger element of chance. It is interesting 
to observe that the two highest marks, 100, are (each) 
more than double the lowest mark, 45. An occur- 
rence of this kind is not at all uncommon in the 

1 ThI Roman numerals express the number of awards of 
marks of which the amounts lie between the Arabic figures 
on either side of them. Thus there were two marks below 
63*5 ; five between 63'5 and jys ; seven between 73*5 and 
78 5, and so on* 

2 The arithmetic mean is 78 (Prof. E.'s footnote). 
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marking advanced work, where there is foom for 
great diversity of taste in the examiners.^' ^ 

, 16 . Prof. E. deduces (with the help of the results 
set forth in § 3 above) that ''in an examination of 
one subject admitting as much diversity of judgment 
asjthe Latin prose ... a difference of 25 per cent, 
between the marks of two candidates may be acci- 
dental and liable to reversal. Suppose, for instance, 
that, under one examiner, candidate A. obtains 90, 
and canti'ldate B. 70, that is, a difference of 25 per 
cent, of the avertige between the two marks, namely 

' No doubt the extremer divergencies are partly due to a 
difference, not of taste, but of scale. In the case of fenglish 
composition and the speculative sciences, I have met with 
equally great discrepancies in the ^narks given by two 
examiners to a particular candidate, without any corre- 
sponding difference in the average of their marks given to a 
whole set of candidates. (Footnote by Prof. E.) 

Prof. E. now adds the following interesting note in re- 
gard to this experiment : — " As to the Latin prose, I*ought 
perhaps to explain that in one respect it was hardly typical. 
Having written it as well as I could, I deliberately dashed it 
with an elementary error, the use of the preposition penes 
with the which, as far as I know, never occurs. This 

schoolboy error, no doubt, as I inffended, utterly puzzled 
the judges. One broad-minded man, now the Provost of a 
distinguished College, said that such a mistake in such a 
paper might be treated as a mere lapsus pluma, and he 
gave it full marks. Another, I think the one who marked 
45, thought the mistake was unpardonable.’* 

According to my own experience, Professor E.'s inten- 
tional mistake increases rather than diminishes the value of 
his experiment. A friend of mine who was pvoxime accessii 
for either the Hertford or Ireland, was told by the examiners 
that except for one thing he would have been a\^arded the 
prize ; he had made one unpardonable mistake in a quan- 
tity in a Latin verse. 
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8o . . . ^ is ijot a very improbabk' event —say u.bouf 
as unlikdy as throwing heads at pitch-fai:thiT?,g six 
times running [sa^^ i to too] — under t.fioth(;r 
examiner the order of the Candida os might he re- 
versed, B coming out above, though probably not high 
above, his rivaj. ' , 

17 . Error of A^%re^ait of Marks formed by Addition 
of Total 1 fo^ different Subjects. — It is necessary to 
deal with aggregates of maiks f'^rm ;d oy the addition 
of totals for diherent subjects for whicb*^e laws 
of error arc differeiit , but Professi^r E. states that 
“ When a coin po and mark is formed by the addition 
of several marks on different subjects, the manner 
of divergence, or law of error, appertaining to the 
compound is of one and the same general type, 
whatever the type of the particular laws appertain- 
ing to the several components. For instance, let 
the several subjects be geometry, history, LaAin prose, 
elc., and let the law of error proper to each of these 
subjects be obtained by*observations such as those 
which have been adduced in the preceding pages. 
Then the compound (mark in geometry + mark in 
history -f mark in Latin prose etc.) will vary with 
the change of exammers according to a law of error 
not in general identical with that of the components, 
but of a certain normal universal type which may be, 
and is often, called the law of error.'" 

Prof. E. states that what is true of sums holds 
good for differences, and this result is used in the 
last paragraph of § 16 above. 

18 . General Case of Distinction awarded for Aggre- 
gates of JS/larks in different Subjects — Use of Results 
obtained front two Markings of the same Work by the 
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t 

Examiner at different Times. — Prof. E. iickW deals 
with the general case in which distinction ^ is deter- 
mined by the snip of marks in different subjects. 

He suggests that if we had to depend on data 
yielded by marking of the same work by several 
examiners, we might despair of obtaining suiB&cient 
statistical material for the reasoning, but thinks it 
suffices to combine a few instances of this type with 
a great many instances of two examiners both mark- 
ing seveiPritl pieces of work. 

Prof, E. invesKgated the results of examination by 
fhe same examiner (Mrs. Bryant, D.Sc.) of the same 
forty papers written by her pupils, on two occasions 
separated by a considerable interval of time. He 
does not give the result directly but states that if 
the same candidate were examined in ten papers by 
two sets of ten examiners, and if each couple of 
examiners “ marked on the whole at the same level 
but with respect to particular papers disagree^with 
each other about as much as IVIrs. Bryant witli her- 
self, then the difference between their marks would 
in the long run as often as not amount to 2*5 per cent, 
(of the average mark for the whole set of ten papers 
examined).'’ 

Thus, if a candidate obtains 2*5 per cent, of the 
average mark for the whole set of candidates above 
the mark for the Honours line, it is as likely as not 
that under another set of examiners he would have 
come out either at least as high as five per cent.* above 
the line, or at least as low as the line. It is with the 
latter alternative only that we are concerned. The 

f 

^ Further details are given in the paper by Prof. E. in the 
Philosophical Magazine for August, 1890. 
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probability of this is oiily half a half, is, a 
quarter/*^ ** 

If a man clears the Honours im^ t)y to per cent, 
of the average marks, the ofids are 300 to 1 against 
a reversal by a different of e^aaiiners : if he clears 
them hy 9 5 per cent , the odds are 200 to i agamst 
reversal ; if he cleais them by 7 per cent, they are 
only 50 CO X. 

In the case of the award of appointments it is, as 
a rule- not an Honours line that is fixed iffa&vance, 
but the number of appointments av\?a.rded. Prof. 
assumes that the difference of marking between 
examkiers will be not^reater than that between the 
two markings by Jiffs. Bryant of the same papers 
on different occasions, and ap^dies this to the case 
of an examination for fifty clerkships in 1875, for 
which there were 172 candidates. He comes to the 
conclusion that there are only 17 candidates for 
whom the odds against being displaced if there were 
a change of examiners is^s great as 50 to i. 

19 . Details in regard to a Classical Tripos Examina- 
tion. — Prof. E. discusses a series of three sets of 
marks given by thre;^ examiners to 75 papers in com- 
position. The average of two of the examiners was 
about the same, but the average of the third examiner 
was about J less than this and was increased there- 
fore by 20 per cent, for purposes of comparison. 

‘ In other words it is equs,! odds whether he is within 
the limit of 5 per cent, above Honours, or outside ; it is 
4 to I against his being outside the limits below the lower 
line, and it is 4 to i against his being outside them above 
the upper •line. If a large number of cases were taken, one 
half would fall*within the limits, one quarter below them, 
and one quarter above them. . 
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The coe%ient of divergence might then be ^pressed 
as a probable error of 5 per cent, on the Sum of 10 
marks.*' • 

“ The severer examinSr is as often as not distanced 
by one of his colleagues to such an extent that one 
mark is double the other. Fourteen times out of 
ninety, one mark is three times as great as another 
given to the same work. Four times, the excess is 
in the ratio 5 to i. And once the severer examiner 
exceed?Rls more lenient colleague in the ratio of 16 to 
These graver discrepancies may no doubt be ob- 
viated by properly correcting the scale of the severer 
examiner. Still the effect of the difference in scale, 
and the efficiency of correction, may easily be exag- 
gerated. It may surprise even (Experts to hear that, 
as between two examiners, A and B, who upon the 
whole are fairly well abreast, occasionally A is ahead 
of B, and B ahead of A to tlie extent of 50 per gent." 
(See also § 11.) 

20. Details in regard to Oxford Literce Humaniores 
Examination in Philosophy. — Prof. E. finds the 
uncertainty in the marking at the Cambridge Classical 
Tripos to be about the same as tjie uncertainty in the 
marking in Philosophy at the Oxford Final Honours 
in Literce. Humaniores, of which he gives some details. 
He points out that the fluctuation in the estimates of 
those who are in or near t^ie first class is markedly 
less than for less successful candidates. “ In fact,'' 
he says, “it is as smalFas any which I have ob- 
served in the case of subjects more elementary and 
precise “ ; a result which he regards as a striking 
tribute to the Oxford method of markjng. < 

21. Details of Examinations in English Composition 
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and Sphulalive Science, — Prof, E. next 'n vestas tes 
the mark!* allotted two different examiners ♦o 400 
"pieces^of English composition. The ''mean ^;rror^' 
(f.<5. the squaie root of the smm of IL ' squares c>f the 
400 differences between the indmdad,! pa'jrs of marks, 
divided by 67 mark<i.i If, instead of taking 

the whole 400 observations, they are divided into 
three batches of 133, the corresponding figures are 
63, 67, and 72. 

The author next points out that tneie differ- 
ence of " scale ' between the tw# sets of marks 
amounting to 10 per cent, of the a^^erage mark. Re- 
ducing the marks to the same scale the “ mean error 
is only reduced to 64 marks. 

The average of the 800 mark<i for the whole set of 
papers then becomes 227 ; and the “ mean error " 64 
is about 28 per cent, of this average. If there were 
ten papers of this kind, the "mean error " for the 

ten would be per cent., and the "probable 
V 10 ♦ 

error " would be 6 per cent. 

Thus, on the aggregate marks for the papers for 
each of which there wa? a " mean error " of 28 per 
cent., the error as likely as not to occur would be 
6 per cent, of the average mark. In a public examina- 
tion on speculative science not guarded so carefully 
as the Oxford Examination in Philosophy," the 
probable error for a set of ten papers amounted to 
8 per ctint. 

22. Summary of Previous Results,— The author now 
sums up his results in the following table of the dis- 
crepancieii as likely as not to occur between the sums 

^ Cf. footnote on pp. 104-5 above. 
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of marks^on ten papers assigned by two sets of ten 
examiners to the same papers. "'"The reader will 
recollect that eadh of these coefficients requires to be 
multiplied by at least 3*5 or 4 in order to obtain the 
height above the Honour line at which a candidate’s 
‘ position may be regarded as irreversible.” 

Number of Probable error calculated for 
papers a set of lo such papers, ex- 
scrutinised. pressed as percentage of the 


High Sc^iool, 6th Form, 
Geometry. (Maiked trvice 
by the same examiner at 
difierent times [see § 18]) 

82 

average aggregate mark. 

2'5 

High School, 6th Form, 
Geometry and History 

160 

3'0 

Cambridge Classical Tripos, 



Composition and transla- 
tion, History and Grammar 
(mixed) (see § 19 ) . 

480 

4 

Latin prose of the type set at 
the I.C.S. Examination 
(see § 16 ) 

' 28 

4*5 

Oxford Literae Humaniores, 
Philosophy only (see § 20 ) 

460 

4*5 

Cambridge Classical Tripos, 
Composition only (see § 19 ) 

225. 

5 

English Composition (see § 

21) .... 

800 

6 

Speculative Science (see § 21 ) 

140 

8 


23 . Question of M easure for Element of Chance in any 
Public Examination, — Prof. E., ” inclining to ^;he side 
of moderation/' adopts as the probable error for marks 
at public examinations the lowest based on the com- 
parison of results of different examiners, viz. 3 per 
cent, of the average mark obtained at the^ examina- 
tion. 
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“^Adopting this datum/' he writes, f an: able 
to answer t]ie followirg questioii vd4 respeipt 
to* any examination of which marks a.e pub-, 
lished : How inany of the candi Jates placed, are 
quite safe, in this sense, thal, if then work had 
been marked by another equ<illy competent 
of examiners, the failure of any assigned one. of 
the successful is a very unlikely event, as un- 
likely as getting a run of ^ix rr ^even heads or 
tails at pitch-farthing ? ^ The public*^niina- 
tions in which I have sought the answer to this 
question fall into three categories: the Indiif 
Civil Service, the Army, and the Home Civii 
Service Clerkships of the second order. I find 
the element of chance in these public examina- 
tions to be such that only a fraction — from a 
third to two -thirds — of the successful candidates 
can be regarded as quite safe, above the danger 
of coming out unsuccessful if a set of equally 
competent examiners, happened to be appointed. 
A corresponding proportion of the successful — 
from two-thirds to one-third — ^must be described 
as unsafe. A rather larger number — ^though 
much smaller proportion — of the unsuccessful 
candidates would have a chance of succeeding at 
a re-examination ; for instance, at three Clerk- 
ships Examinations, 25, 43, 72, out of a total 
number of unsuccessful, respectively, 126, 102, 
and 121, • 

" This question may also be asked : What 

1 Odds of 100 to I, corresponding to a height above the 
Honour lime of 3’5 times the " probable discrepancy." (Foot- 
note by Prof. E*.) 

K 


129 



EX^MINATIONS-NQTE E . 

number of displacements is most likely do occur 
in thfc event of a re-examination ? The answers 
in some cas^ which I have scrutinised proves to 
be about a seventh part of the successful. Thus, 
where fifty clerks were appointed, it may be 
expected — the most probable event is — that seven 
would be displaced by a re-examination of the 
same work." 

Prof-®. gives here an elaborate table relating to 
the results of a*' number of Civil Service and Army 
Examinations for which reference must be made to 
the original. 

Prof. E. comes to the conclusion that the distinc- 
tion between adjacent candidates at such examina- 
tions as arranged in accordance with the marks 
allotted is in hardly any case a real distinction. 
** Differences of marks on the strength of which one 
Civil Servant is promoted ^ above his fellow correspond 
to only the faintest probability of real superiority. 
Occasionally, but by no means invariably — in exa- 
minations decided by an aggregate of marks — ^the 
senior or the first two or three break away from their 
neighbours by an interval which cannot be ascribed 
to accident. But between the adjacent men among 
the rank and file the distii\ction is hardly ever real." 

84 . Commeyits by Prof. E. on his Results. — Prof. E. 
thinks his results err, if anything, on the side of 
moderation, for the following reasons : (i) The cor- 
rection for mutually compensating errors (see 

i The author, by “ promoted,” means here y placed at 
the entrance examination ” above another \;andidate, being 
thus entitled to a superior initial position in the service. 
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§ 8) is probably an o^’^er-coirection, and then !s *’ an 
intractably miniflium of constant errci (2) the 

divisioti used for this correction, is too grea| 
for some examinations, the arerage u amber papers 
being probably nearer sevei: than ten ; (3) the assump- 
tion that differeTices of scale of niarkirg (see § 7) ^’e 
corrected, or car:, be allowed for is *' precarious," 
smce candidates do not alw'iys take the same sub- 
jects ; (4) owing to change^ of ...caie, the hypothesis 
that the Honour line may be regarded aspucollstant, 
whatever the change of examiners^ does not hold. 
Hence the error is greater than that shown. * 

On ^the other hand, the nneertaint}^ about high 
marks appears to be less than about lower ones ; 
in order to obtain the best resvHs we should consider 
only those above the line of what may be regarded as 
certain failure. 1 " Possibly the uncertainty of some 

particular examinations may have been exaggerated." 

25 . Question of making M arks of a given Examination 
conform to a given Freqmncy Curve [see § 16 (3)]. — 
Prof. E, suggeststhat any hard- and fast rule would be 
dangerous. Referring presumably to Government 
examinations, he says: 

" Perhaps the best general principle is that 
the ‘ quartile ’ and ‘ median ’ should be fixed for 


^ It seems clear a priori that examiners will not take as 
much trouble in regard to candidates whom they regard as 
unlikely to succeed. I do not know whether, at the date 
of the examinations investigated by Prof. E., the Civil Service 
Examiners made the " deduction for smattering " of which 
the existence has long been known, but which was first 
officially described in the Report of Mr. Stanley Leathes* 
Committee (see p. 17). 
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each class of examination by a Committee of 
pembneiit officials, in the lighj of an ^ extensive 
experience in pari materia ; that the examiner 
on each occasion should be expected to 
furnish a set of marks roughly conforming to 
the pattern thus prescribed ; and that any 
adjustment which the marks furnished may seem 
to require, should not be performed without the 
concurrence of the acting examiner, who has scru- 
tiftised the merit of each particular candidate." 

K 26. Inevitability of an Element of Chance — Sugges- 
tions for the Future. — Professor E. points out that there 
is an incorrigible minimum of uncertainty, and is 
satisfied that in some of the cases investigated that 
minimum could not be substantially reduced by any 
method of correction. 

For hardship entailed on unsuccessful candidates 
excluded by chance he says : " Perhaps here, as in 
other grievances, mere publicity is a sufficient, and the 
sole obtainable, remedy. The general recognition of 
the element of chance in examinations would miti- 
gate the disappointment of those who, for want of a 
few marks, are either altogether excluded from the 
class of the successful, or compelled without prefer- 
ment to take a low place in that class." 

Prof. E. suggests, rather lor the sake of facilitating 
the conception of his theory “ than as a serious prac- 
tical proposal," that candidates within the limits of 
uncertainty should draw lots for the vacant"* places 
allotable, each candidate receiving a number of tickets 
proportional to the chance that he is really deserving, 
as accurately ascertained by the theory ©f proba- 
bilities. He points out that such a proposal would 
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mitigate the sense of injustice in disapp oinked indi- 
viduals and d<\ more to bring home to fhe public 
‘mind The aleatory character of e -^minations than 
even an article in the of the Royal Statistkcil 

Sociely** 

Prof. E. point.’ out the element of certainty as ^eU 
as that of chance in public oearainaticns. He coin- 
pares their lesuits to these of a barometer uncorrected 
for violent changes of teraperat 1 1 o . ‘ That exami na- 
tions are a very rough test of merit, but th%’ dfl serve 
to distinguish excellence in certain qualities, is a con- 
clusion which recommends itself to practical anfi 
sensible persons/' 

But ‘ meie common sense, without the calculus of 
probabilities, is nolt competen" adequately to define 
the limits of chance and certainty. Common sense 
might be embarrassed in defending its position against 
the awkward tact which has been adduced, that even 
in the best regulated examinations one examiner 
occasionally differs from ^another to the extent of 50 
per cent. ; for common sense cannot fully appreciate 
that mutual compensation of errors in virtue of 
which a peculiar stability attaches to aggregates of 
marks.'’ 

• In a charming peroration (so suave one hardly dare 
suspect it of irony) Prof. E. suggests that “To 
examiners at least it will be interesting to test the 
accuracy of the instrument with which they work. 
The statistical study may beguile the monotony of 
their task." 

27 . Conclusion . — In presenting the above summary 
of Professor Edgeworth's papers I must disclaim once 
more any pretension to be an authority on statistics 
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or to add by quotation any weight to Prof. Edge- 
worth's results. They are those ofjU recognised ex- 
pert to whom oqe cannot but be grateful. Ih spite ' 
of the soothing remarks'which I have. I hope, repro- 
duced as faithfully as the disturbing figures, the 
papers leave me, and I believe they will leave others, 
with the conviction that the necessity for further 
investigation is real and pressing. Prof. Edgeworth’s 
results, significant as they are, are based on data 
some iSiii^y years old. What are the facts now ? 
Ought not each examination on a large scale to be 
jferiodically subjected to the criticism of a professional 
statistician ? We cannot avoid the effects of chance 
in examinations — such chance exists in all human 
affairs — ^but it is surely a duty both to ascertain what 
they are, and to reduce them to a minimum. 

And I repeat once more my belief that, if examiners 
asked themselves exactly how they could define in 
words what the passing of an examination means, 
so as to be able to give a definite assurance to the 
public in regard to what all the successful candidates 
can do, the element of chance would be singu- 
larly reduced. 

I make one last quotation from Prof. Edgeworth, 
which is appropriate here though he uses it in a some-* 
what different connection : ^ 

" What should we think of a company who, 
dealing in thousands and millions sterling, 
grudged the payment* of a professional account- 
ant to check the accounts of their Directors ? 
It would be a similarly false economy on the part 
of educated society, considering the stijpendous 
sum of mental labour involved in the system 

134 



EXAMIN AT JC^S— E 

of competitive exammationa, to gt the cx- 
pen4ilture of a little additional etfc:t fix a tech- 
nique which professes to bring^o book, i'vA test 
the figures of the examining bodios who a:?e 
responsible for the direction of that labour/' 



NOTE F 

ON KNOWLEDGE-TESTS AND MEMORY-TESTS 


It is frequently impossible in a written examination 
to distinguish knowledge, implying understanding, 
from pure^emory. The instance of the candidate 
who, in translating a passage from a prepared book, 
cfx,rries his translation beyond the passage set, illus- 
trates the point. 

It has been suggested by a critic that I have Qver- 
estimated the importance attached to memory-tests 
in examinations at the present time. I may point 
out in this connection that Prof. Burnet, in his Higher 
Education and the War, strongly advocates the use 
of set books, and deprecates the use of unseen trans- 
lation in classical teaching in schools. I am not 
prepared to contest the soundness of his views in this 
particular matter ; I quote them as showing that 
memory-tests are not obsolete, and that precautions 
should be taken in using them. ‘ 

Since the publication of the above criticism, one 
extremely successful candidate told me that, having 
to pass examinations in English (which, although he 
is a British subject, is not his mother-tongue) he learnt 
off by heart answers which ^he did not understand to 
questions which he recognised by catch-words, and 
passed a series of examinations in this way. He 
assured me that his case was an extremely common 
one in his own part of the British donfinioils. An 
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Englishwoman who was first or second in each of the 
four subjects of a first-year Universit}^ ^.curse in 
natural science at cne of the Uiiiversi|5’' 

colleges in Great Britaiii a^ew years since, has also 
assured me that in physics, in which she took tho 
second pi?^ce, she would have been quite incapaj^le 
of explaining some of her oVvH answers ; and that her 
personal success gave her a rooted disbelief in the 
results of examinations as .now conducted. 

Charles Lister, while serving his diplomatic appren- 
ticeship at Rome in 1912, co’umenfed on the bliss- 
ful ignorance o.t commerce in the diplomatic servi?;e. 
(now, about to be remedied), and v/rote: I should 
suggest some compulsory subject in the examina- 
tions in banking, finance, and irade, probably 
some Hawkins would come along and put it all into 
a book in the form most easily learnt and most easily 
forgotten, and little would come of it {Charles 
Lister : Letters and Recollectio?is , 1917, p. 45). In the 
considerable literature on examinations the views of 
the recently examined are comparatively rare ; I 
attach importance to them, especially in the case of 
able candidates such as those above cited. My 
critic, M. H. (see Times Educational Supplement 
Ifor August 2, 16, and 23, 1917) writes : “ Only after- 
life can show how far the acquirements of memory 
aie fleeting : unconscious stores may play their part 
in the intellect ; it is better to have learned and 
lost than never to have learned." The excellent 
epigram seems to me applicable only to knowledge 
which has been assimilated. The examples quoted • 
above show that unassimilated knowledge may and 
does stifl seiVe its turn in the examination- room. 
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ON THE USE OF THE TERM ORDER 
OF MERIT 

I HAVE consulted? my friend Prof. S, Alexander, on 
tie significance attached to the word merit,*’ 
as used by professional writers on ethics ; he has 
kindly referred me to Leslie Stephen’s Science of 
Ethics, ch. vii., § i Merit, and § 2 Effort ; James 
Martineau’s Types of Ethical Theory, VoL II., Book I, 
ch, iii., Merit and Demerit ; and to his own Moral 
Order and Progress. In the last-named work (p. 194), 
Prof. Alexander writes of merit : “ Though in reality 
a simple idea, it has given H’ise to the greatest con- 
troversy, from the apparent divergence with which 
it is applied.” This is not the place to discuss 
general theories of ethics, even if the writer were 
competent to do so. But the objection to the use 
of the word merit ” in the phrase ” order of merit 
is that it really leads to confusion of ideas, as indi- 
cated in § 17 , p. 63 above. I find myself in modest 
disagreement with Prof. Alexander's reconciliation of 
different views, and in general sympathy witfi Mar- 
tineau’s interpretation, which Prof. Alexander thinks 
is also that of ” the man in the street.” 

I think that confusion arises frequentl}^ because 
in using the word one is apt, when speakmg of an 
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action, to lose the distinction l>etween («) the intrinsic 

merit of 4he acMon considered Indeoendeift)} of the 
• * 
author, and (5) th^s merit involved i> its perloiman^:^. 

The more dazzling the action, the rnorr dlfBcult d<»sa 
the public find it to make the distinction. A chanc€ 
obser^jativM'i of scientific importance may confer un- 
d^dng fame on the observer. Heroism may confei 
immortalitv^ on a failure. 

But the leality of the distinction will made 
clear by a concreie hypothetical example connected 
with onr subject. Suppose you aTe shown a short 
piece of French prose, irreproachable in grammar aud 
‘‘ quite good " in style, of w'hich you do not know 
the uuthor. You say, "'This piece has merit.'" You 
then ask by whom ft was written, and are told that it 
was written (say) by Anatole France. “ Ah," you 
say, " not up to his usual level." You are then 
informed that the previous information was mistaken, 
and that the piece was by an English boy who has 
never been in France ; aftd you are filled with admira- 
tion. Clearly, one attributes great merit to the 
English schoolboy for a performance for which one 
would attribute npne to Anatole France, yet the 
value of the piece of prose itself remains the same. 

* The example is not so far removed from reality as 
it may seem. While it is quite true that Frencl: 
authors do not enter for our examinations in French, 
candidates who have been partly trained in France 
do so* and compete agadnst others who have con- 
tended against the greatest possible difficulties, it 
may be both of natural disposition and of external 
opportu|iity, in order to achieve the same result 
as those more favoured by nature and by fortune 
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Clearly we have no right to say, then, that examina- 
tions pladfe the candidates thetnsd'ms in order of 
n^erit.” But we^ught to be justified in sayirig that 
examiners place candidates in the order of proficiency 
shown by their answers, which is another matter. 

Jo the philosophical critic, I venture to point out 
that the case is simplified by the fact that exami- 
nation answers are not (in general) productions so 
dazzling that all distinction between the merit of -thfe 
author and of the production is perforce obliterated 
in the public eye kad understanding. Their intrinsic 
public value, except in a few delightful instances, is 
nought. Having served their purpose, they «pass, 
unregretted, to the oblivion of the paper-mill. • 
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Mistake, single, varying import- 
ance attached to by 
examiners, 122 
Muench, W., 99 
MUnsterberg, 77 

Natural ability, 84, 88 
Negative marks, 61 
Negligence of haste, errors due 
to, no, in, 116, 117 
Nervousness in examinations, 

■ 20, 21, 68 

Newman, John Henry, 8, 93 
Nomination for appointments, 
37-8 

Nunn, Prof. J. P., xviii, 71 


Objects of examination, 3-1 1, 
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